CN115599704A - File system metadata separate storage method and device and storage medium - Google Patents
File system metadata separate storage method and device and storage medium Download PDFInfo
- Publication number
- CN115599704A CN115599704A CN202211519205.1A CN202211519205A CN115599704A CN 115599704 A CN115599704 A CN 115599704A CN 202211519205 A CN202211519205 A CN 202211519205A CN 115599704 A CN115599704 A CN 115599704A
- Authority
- CN
- China
- Prior art keywords
- request
- metadata
- mapping
- user data
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0664—Virtualisation aspects at device level, e.g. emulation of a storage device or system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1016—Performance improvement
- G06F2212/1024—Latency reduction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a device and a storage medium for separating and storing metadata of a file system, wherein the method comprises the following steps: when a file system generates a metadata IO request, adding a specified tag for metadata; the method comprises the steps of intercepting a metadata IO request and a user data IO request through pre-registered virtual storage equipment, identifying the type of the IO request according to whether a label is attached or not when the IO request is intercepted by the virtual storage equipment, carrying out address mapping according to the type of the IO request and redirecting the request to corresponding storage equipment, wherein if the request is the metadata IO request, the request is redirected to the metadata storage equipment, and if the request is the user data, the request is redirected to the user data storage equipment. The method has the advantages of simple implementation method, low cost, high metadata access efficiency, no influence of metadata operation on the continuous IO requests of the user data and the like.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a method and a device for separately storing metadata of a file system and a storage medium.
Background
In a storage system, physical storage space is managed by a file system. In order to implement the space management function, the file system needs to record information such as the distribution of user data on a disk, the size of the user data, and modification time, in addition to storing the user data, and these auxiliary recording information are metadata.
Metadata is typically small and discrete with respect to user data. Common file systems, such as xfs, ext3/4, btrfs, etc., are managed by mixing and recording user data and metadata on the same storage device, as shown in fig. 1. The above-described metadata storage method has the following problems:
1. mass storage systems typically employ mechanical disks as storage devices. For metadata information distributed discretely on a disk, a head needs to be moved frequently to perform a seek operation during access, so that metadata access efficiency is low.
2. For user data IO which is simultaneously in progress, originally continuous and large number of IO operations can be interrupted by fragmented metadata requests, and the performance of the whole system is influenced.
If the metadata separation is directly performed in the file system, a multi-storage device management function needs to be added to the file system, and the metadata and the user data are respectively stored in different devices, which requires structural adjustment of the file system, and therefore the file system is high in development difficulty, large in workload, and difficult to ensure reliability. Therefore, it is desirable to provide a method for implementing separate storage of metadata of a file system, while reducing implementation complexity and minimizing the adjustment of the file system itself.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: aiming at the technical problems in the prior art, the invention provides a file system metadata separation storage method, a device and a storage medium, wherein the file system metadata separation storage method, the device and the storage medium have the advantages of simple realization method, low cost, no need of structural adjustment on a file system, high metadata access efficiency and no influence of metadata operation on continuous IO (input/output) requests of user data.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
a file system metadata separation storage method includes the following steps:
when a file system generates a metadata IO request, adding a specified tag for metadata;
intercepting a metadata IO request and a user data IO request through pre-registered virtual storage equipment;
when the IO request is intercepted, the virtual storage device identifies the IO request type according to whether the tag is carried, performs address mapping according to the IO request type and redirects the request to the corresponding storage device, wherein if the IO request is metadata IO request, the request is redirected to the metadata storage device, and if the IO request is user data request, the request is redirected to the user data storage device.
Further, if the request is a metadata IO request, the step of performing address mapping includes:
searching a pre-established metadata mapping record tree, and if a mapping node corresponding to the current metadata IO request is searched, modifying the address of the current IO request into a corresponding mapping address according to a search result, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information;
and if the corresponding mapping node is not searched in the metadata mapping record tree, searching a pre-established user data mapping record tree, distributing a metadata mapping address according to whether the mapping node corresponding to the current metadata IO request is searched, and modifying the address of the current metadata IO request into the corresponding distributed mapping address, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information.
Further, the allocating the metadata mapping address according to whether the corresponding mapping node is searched comprises:
and judging whether a corresponding mapping node is searched in the user data mapping record tree or not, if so, releasing the user data mapping record and distributing the metadata mapping address after releasing the tree node of the user data mapping record, if not, directly distributing the metadata mapping address, and updating the metadata mapping record file after distributing the metadata mapping address.
Further, if the request is a user IO request, the step of performing address mapping includes:
searching a pre-established user data mapping record tree, and if a mapping node corresponding to the current user IO request is searched, modifying the address of the user IO request into a corresponding mapping address according to a search result, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information;
if the mapping node corresponding to the current user IO request is not searched in the user data mapping record tree, searching a metadata mapping record tree, distributing a user data mapping address according to whether the mapping node corresponding to the current user IO request is searched, and modifying the address of the current user data IO request into the corresponding distributed mapping address, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information.
Further, the allocating the user data mapping address according to whether the mapping node corresponding to the current user IO request is searched includes:
and judging whether a corresponding mapping node is searched in the metadata mapping record tree or not, if so, releasing the metadata mapping record and distributing the user data mapping address after releasing the metadata mapping record tree node, if not, directly distributing the user data mapping address, and updating the user data mapping record file after distributing the user data mapping address.
Further, the mapping record tree is a balanced binary tree with the virtual device disk offset as an index, where each node includes the virtual device disk offset, the data length, the physical memory offset, the data mapping record number, the parent node, and the child node information.
Furthermore, before intercepting the metadata IO request and the user data IO request through the pre-registered virtual storage device, the method further includes starting the virtual storage device, reading physical device information from the configuration file, and after loading the data mapping record to the virtual storage device, respectively establishing a metadata mapping record tree and a user data mapping record tree, wherein the metadata mapping record tree includes the metadata mapping record and corresponding parent-child node information, and the user data mapping record tree includes the user data mapping record and corresponding parent-child node information.
Further, the virtual storage device is pre-registered on the generic block device layer, and a mapping driver of the virtual storage device has a function of loading a preset mapping rule from a specified location.
A file system metadata split storage comprising:
the tag adding module is used for adding a designated tag for the metadata when the file system generates a metadata IO request;
and the virtual storage equipment is used for intercepting the metadata IO request and the user data IO request, identifying the type of the IO request according to whether the tag is attached or not when the IO request is intercepted, carrying out address mapping according to the type of the IO request and redirecting the request to the corresponding storage equipment, wherein if the request is the metadata IO request, the request is redirected to the metadata storage equipment, and if the request is the user data request, the request is redirected to the user data storage equipment.
A computer-readable storage medium having stored thereon a computer program which, when executed, implements the method as described above.
Compared with the prior art, the invention has the advantages that: according to the method and the device, when the file system submits the IO request, the specified label is added aiming at the metadata request, the IO request is intercepted and identified by the virtual storage device, the address mapping is carried out according to the type of the IO request, the request is redirected to different storage devices to be separately stored and accessed, the management of real storage devices and the mapping and forwarding of the IO request are realized, the separate storage of the metadata can be realized under the condition that the program of the file system is modified as little as possible, the continuous IO request of user data is not influenced by metadata operation, the metadata access speed is greatly improved, and the overall operation performance is improved.
Drawings
Fig. 1 is a schematic diagram illustrating a storage principle of metadata and user data in a conventional file system.
Fig. 2 is a schematic flow chart of an implementation of the file system metadata separate storage method according to the embodiment.
Fig. 3 is a schematic structural diagram of a binary tree of data mapping records established in this embodiment.
Fig. 4 is a schematic diagram of a data structure of a storage device and an available space management mechanism in this embodiment, where (a) represents a disk space configuration of the storage device, and (b) represents a specific representation of a space bitmap.
Fig. 5 is a schematic flowchart of virtual block device loading in this embodiment.
FIG. 6 is a schematic diagram illustrating a principle of intercepting a BIO request in an embodiment of the present invention.
Fig. 7 is a schematic flowchart of a detailed process for implementing separate storage of metadata of a file system in an embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the drawings and the specific preferred embodiments, without thereby limiting the scope of protection of the invention.
As shown in fig. 2, the steps of the file system metadata separate storage method of this embodiment include:
s01, when a file system generates a metadata IO request, adding a designated tag for metadata;
s02, intercepting a metadata IO request and a user data IO request through a pre-registered virtual storage device, identifying the IO request type according to whether a label is attached or not when the virtual storage device intercepts the IO request, performing address mapping according to the IO request type and redirecting the request to a corresponding storage device, wherein if the request is the metadata IO request, the request is redirected to the metadata storage device, and if the request is the user data request, the request is redirected to the user data storage device.
In the embodiment, a virtual storage device is added between a file system and a real storage device, when the file system submits an IO request, a specified tag is added for a metadata request, the IO request is intercepted and identified by the virtual storage device (a driver), address mapping is performed according to the type of the IO request, and the request is redirected to different storage devices to be separately stored and accessed, so that management of the real storage device and mapping and forwarding of the IO request are realized, that is, the virtual storage device hides a specific physical device from the file system upwards, and redirects the IO request of the metadata and the IO request of user data downwards to different storage devices, so that metadata and user data are sent to different devices, separate storage of the metadata can be realized under the condition that the file system program is modified as little as possible, continuous IO requests of the user data are not affected by metadata operation, meanwhile, the metadata access speed is greatly increased, and the overall operation performance is improved.
The mapping of metadata operations to the smaller space of high speed memory by the driver of the virtual storage device performs IO.
The metadata storage device may be a high-performance solid-state storage device, and the user data storage device may be a large-capacity mechanical storage device, so that the metadata operation may be mapped to a high-speed storage in a smaller space to perform IO, and the user data operation is performed by the large-capacity mechanical storage device, which balances performance and cost of data storage implementation.
In step S01 of this embodiment, a file modification system is specifically adjusted to attach a metadata tag to a private field of a request when a metadata IO request is generated, the user data IO request remains as it is without modification, and a driver of the virtual storage device can identify whether the request is a metadata IO request or a user data BIO request according to the tag in a subsequent processing flow.
The virtual storage device of this embodiment may specifically be implemented by using a virtual block device (block device), where a mapping driver of the virtual storage device has a function of loading a preset mapping rule from a specified location, and the virtual block device is used to intercept and redirect metadata and user data to different storage devices, and the virtual storage device is registered in advance on a general block device layer. A virtual storage device is added between a file system and a real storage device, the file system sends all Block device IO requests (BIO requests) to the virtual Block device, the virtual Block device receives metadata BIO requests and user data BIO requests from the file system, the virtual storage device replaces the real storage device in the traditional process, a uniform address space is provided for the file system, and the real storage device forming the space is managed. The virtual block device driver may redirect the metadata IO request to a desired metadata storage device (e.g., a small-capacity high-speed solid-state storage device) and direct the user data IO request to a desired user data storage device (e.g., a large-capacity mechanical disk array) according to a configuration set by a user.
In this embodiment, intercepting and redirecting the metadata and user data IO request through the virtual block device further includes starting the virtual storage device, reading physical device information from the configuration file, and loading the data mapping record to the virtual storage device to respectively establish a metadata mapping record tree and a user data mapping record tree, where the metadata mapping record tree includes the metadata mapping record and corresponding parent-child node information, and the user data mapping record tree includes the user data mapping record and corresponding parent-child node information, where the parent-child node information includes parent node information, left child node information, right child node information, and the like.
As shown in fig. 3, the data mapping tree (metadata mapping record tree, user data mapping record tree) is a balanced binary tree with the virtual device disk offset as an index, and is used for quickly searching for mapping records according to the virtual device offset, and each node in the data mapping tree includes the virtual device disk offset, data length, physical memory offset, data mapping record number, parent node, left child node, right child node information, and the like.
As shown in table 1, in the specific application embodiment, each record in the data mapping record file has a fixed length, and includes a virtual device disk offset, a data length, a data storage offset, data mapping record number information, and the like, that is, a storage form of the mapping record on the disk.
Table 1: data mapping record table
| Data mapping record number | Virtual device offset | Data length | Data memory offset |
| 0 | 347136 | 4096 | 0 |
| 1 | 321536 | 1024 | 4096 |
| 2 | 470016 | 2048 | 8192 |
| …… | …… | …… | …… |
| 1024 | 33554432 | 512 | 524288 |
As shown in fig. 4, where (a) in fig. 4 represents the configuration of the disk space of the storage device, and (b) in fig. 4 represents a specific representation of the space bitmap, in this embodiment, two fixed disk spaces are allocated at the front ends of the metadata storage device and the user data storage device, and are respectively used for storing the mapping record and the space bitmap, so as to implement management of the space in the form of a bitmap. The space bitmap specifically represents 4096 bytes of storage space with 1 binary bit, a value of 0 indicates that space is unused, and a value of 1 indicates that space is used.
In a specific application embodiment, as shown in fig. 5, the loading process of the virtual block device (virtual storage device) implemented in this embodiment is as follows:
step 1. Virtual block device start-up
And 2, reading the physical equipment information from the configuration file (stored in the operating system configuration file directory), and setting the metadata storage equipment and the user data storage equipment.
And 3, loading the metadata mapping record from the mapping record file of the metadata storage device.
And 4, inserting the metadata mapping record into a metadata mapping binary tree in the memory.
And 5, loading the mapping record of the user data from the mapping record file of the user data storage equipment.
And 6, inserting the user data mapping record into the user data mapping binary tree in the memory.
And 7, finishing the registration operation of the virtual block equipment to the operating system.
The above steps do not limit the time sequence relationship, the sequence between the steps can be adjusted, for example, the binary tree mapping metadata and user data can be established sequentially or simultaneously, and the specific execution time sequence can be configured according to actual requirements.
Because the requests sent by the file system are all in the same address space, and the metadata separation requires that the user data and the metadata are respectively stored in the corresponding devices, the address space needs to be modified, and for the difference in space between the metadata storage device and the user data storage device, the virtual address IO requests from the file system need to be respectively remapped to the address space of the real physical device after being classified according to the metadata and the user data. In step S02 of this embodiment, when the virtual storage device is registered in the general-purpose block device layer, the mapping driver corresponding to the virtual storage device loads the mapping rule defined by the user from the designated location; in the read-write flow, the IO request is identified by starting a mapping driving program, the address of the request is mapped to the address space of the corresponding storage device according to the mapping rule, the metadata IO request is redirected to the metadata storage device by using the new address, and the user data is redirected to the user data storage device.
In this embodiment, if the request is a metadata IO request in step S02, the step of performing address mapping includes:
s211, searching a pre-established metadata mapping record tree, and if a mapping node corresponding to the current metadata IO request is searched, modifying the address of the current IO request into a corresponding mapping address according to a search result, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information;
s212, if the corresponding mapping node is not searched in the metadata mapping record tree, searching a pre-established user data mapping record tree, distributing a metadata mapping address according to whether the mapping node corresponding to the current metadata IO request is searched, and modifying the address of the current metadata IO request into the corresponding distributed mapping address, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information.
The step S212 of allocating the metadata mapping address according to whether the corresponding mapping node is searched specifically includes:
and judging whether a corresponding mapping node is searched in the user data mapping record tree or not, if so, releasing the user data mapping record and distributing the metadata mapping address after releasing the user data mapping record tree node, if not, directly distributing the metadata mapping address, and updating the metadata mapping record file after distributing the metadata mapping address.
For a metadata IO request, if a corresponding mapping node is found in a metadata mapping record tree, the mapping node may be directly modified to be a post-mapping address (specifically, a disk offset of the modification request may be a post-mapping disk offset), if a corresponding mapping node is not found, a user data mapping record tree is further searched, and if no corresponding node still exists, it indicates that a current IO request needs to newly allocate a space, a space that meets the current IO request may be allocated from an available space of a metadata memory; if the node corresponding to the current IO request address is found in the user data mapping binary tree, which indicates that the current request address is used for storing user data, the user data mapping record is released (the corresponding bit of the space bitmap of the user data storage device is marked as available, the record is deleted from the user data mapping file, and the mapping node is deleted from the user data mapping record tree), and then the metadata mapping address is redistributed, so that the address mapping and redirection of the metadata can be realized.
In step S02 of this embodiment, if the request is a user IO request, the step of performing address mapping includes:
s221, searching a pre-established user data mapping record tree, and if a mapping node corresponding to the current user IO request is searched, modifying the address of the user IO request into a corresponding mapping address according to a search result, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information;
s222, if the mapping node corresponding to the current user IO request is not searched in the user data mapping record tree, searching a metadata mapping record tree, distributing a user data mapping address according to whether the mapping node corresponding to the current user IO request is searched, and modifying the address of the current user data IO request into the corresponding distributed mapping address, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information.
The step S222 of allocating the user data mapping address according to whether the mapping node corresponding to the current user IO request is searched includes:
and judging whether a corresponding mapping node is searched in the metadata mapping record tree or not, if so, releasing the metadata mapping record and distributing the user data mapping address after releasing the metadata mapping record tree node, if not, directly distributing the user data mapping address, and updating the user data mapping record file after distributing the user data mapping address.
Specifically, for a user data request, a user data mapping record tree is searched according to a disk starting point offset of the user data request, and if a corresponding mapping node is found, an address of the request is directly modified to be a mapped address (specifically, the disk offset of the request can be modified to be the disk offset after mapping); if the corresponding mapping node is not found, further searching the metadata mapping record tree, if the corresponding node still does not exist, indicating that the current IO request needs to be newly allocated, and allocating a space meeting the current IO request from the available space of the user data memory; if the node corresponding to the current IO request address is found in the metadata mapping record tree, which indicates that the current request address is used for storing metadata, the metadata mapping record is released (the corresponding bit of the metadata storage device space bitmap is marked as available, the record is deleted from the metadata mapping file, and the mapping node is deleted from the metadata mapping record tree), and then the user data mapping address is allocated to the mapping record tree, so that the address mapping and redirection of the user data can be realized.
Through the steps, the location of the real data can be found by the metadata and user data storage device through the virtual storage device address, so that the two types of data can be correctly stored and read.
In a specific application embodiment, as shown in fig. 6, the virtual storage device uses a block device, and sends the metadata BIO request and the user data BIO request of the file system to the virtual block device, as shown in fig. 7, the detailed steps of the virtual cache device driver in the virtual block device implementing mapping of different data addresses by using the above mapping operation of the present invention are as follows:
extracting a metadata flag from private data of a BIO request of an operating system.
And b, if the metadata mark is detected, modifying the field of the target disk and redirecting the BIO request to the metadata storage device.
Address mapping and redirection for metadata requests:
and c, searching the metadata mapping binary tree aiming at the disk starting point offset of the metadata request, and if the corresponding mapping node is found, modifying the disk offset of the request into the disk offset after mapping.
If no corresponding mapping node is found, further searching the user data mapping binary tree, if no corresponding node still exists, indicating that the current BIO request needs to be newly allocated, and allocating a space meeting the current BIO request from the available space of the metadata memory; if the node corresponding to the BIO request address is found in the binary tree of user data mapping, the user data mapping record is released and the mapping node is deleted from the tree of user data mapping records and then the metadata mapping address is allocated to the mapping node.
And e, inserting a new mapping record into the metadata mapping binary tree.
And f, updating the metadata mapping record file.
And g, modifying the disk offset of the BIO request into the mapped disk offset.
Sending a metadata BIO request.
If no metadata tag is detected in step b, modifying the target disk to be the user data storage device.
And j, searching the user data mapping binary tree aiming at the disk starting point offset of the user data request, and if the corresponding mapping node is found, modifying the requested disk offset into the mapped disk offset.
If no corresponding mapping node is found, further searching the metadata mapping binary tree, if no corresponding node still exists, indicating that the current BIO request needs to be newly allocated, and allocating a space meeting the current BIO request from the available space of the user data memory; if the node corresponding to the BIO request address is found in the binary tree of metadata mapping, the metadata mapping record is released and the mapping node is deleted from the tree of metadata mapping records and then assigned the user data mapping address.
And step l, inserting a new mapping record into the user data mapping binary tree.
And m, updating the user data mapping record file.
And n, modifying the disk offset of the BIO request into the mapped disk offset.
And step o, sending a user data BIO request.
The file system metadata separation storage device of the embodiment comprises:
the system comprises a tag adding module, a metadata generating module and a metadata processing module, wherein the tag adding module is used for adding a specified tag for metadata when a metadata BIO request is generated by a file system;
the virtual storage device is used for intercepting the metadata BIO request and the user data BIO request, identifying the type of the BIO request according to whether a label is attached or not when the BIO request is intercepted, carrying out address mapping according to the type of the BIO request and redirecting the request to the corresponding storage device, wherein if the BIO request is the metadata BIO request, the BIO request is redirected to the metadata storage device, and if the BIO request is the user data request, the BIO request is redirected to the user data storage device.
The tag adding module can be realized by adding a tag adding function in the file system, and can realize the separated storage and access of the metadata and the user data only by simply modifying the file system without structurally adjusting the file system.
The file system metadata separation storage device of this embodiment corresponds to the file system metadata separation storage method described above, and is not described herein any more.
The present embodiment also provides a computer-readable storage medium storing a computer program, which when executed implements the method as described above.
The foregoing is considered as illustrative of the preferred embodiments of the invention and is not to be construed as limiting the invention in any way. Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical spirit of the present invention should fall within the protection scope of the technical scheme of the present invention, unless the technical spirit of the present invention departs from the content of the technical scheme of the present invention.
Claims (10)
1. A file system metadata separation storage method is characterized by comprising the following steps:
when a file system generates a metadata IO request, adding a specified tag for metadata;
intercepting a metadata IO request and a user data IO request through pre-registered virtual storage equipment;
when the IO request is intercepted, the virtual storage device identifies the IO request type according to whether the tag is carried, performs address mapping according to the IO request type and redirects the request to the corresponding storage device, wherein the request is redirected to the metadata storage device if the request is the metadata IO request, and the request is redirected to the user data storage device if the request is the user data request.
2. The method for separately storing the metadata of the file system according to claim 1, wherein if the request is a metadata IO request, the step of performing address mapping comprises:
searching a pre-established metadata mapping record tree, and if a mapping node corresponding to the current metadata IO request is searched, modifying the address of the current IO request into a corresponding mapping address according to a search result, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information;
and if the corresponding mapping node is not searched in the metadata mapping record tree, searching a pre-established user data mapping record tree, distributing a metadata mapping address according to whether the mapping node corresponding to the current metadata IO request is searched, and modifying the address of the current metadata IO request into the corresponding distributed mapping address, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information.
3. The method for separately storing metadata of a file system according to claim 2, wherein the allocating metadata mapping addresses according to whether corresponding mapping nodes are searched comprises:
and judging whether a corresponding mapping node is searched in the user data mapping record tree or not, if so, releasing the user data mapping record and distributing the metadata mapping address after releasing the tree node of the user data mapping record, if not, directly distributing the metadata mapping address, and updating the metadata mapping record file after distributing the metadata mapping address.
4. The method for separately storing the metadata of the file system according to claim 1, wherein if the request is a user IO request, the step of performing address mapping comprises:
searching a pre-established user data mapping record tree, and if a mapping node corresponding to the current user IO request is searched, modifying the address of the user IO request into a corresponding mapping address according to a search result, wherein the user data mapping record tree comprises user data mapping records and corresponding parent-child node information;
if the mapping node corresponding to the current user IO request is not searched in the user data mapping record tree, searching a metadata mapping record tree, distributing a user data mapping address according to whether the mapping node corresponding to the current user IO request is searched, and modifying the address of the current user data IO request into the corresponding distributed mapping address, wherein the metadata mapping record tree comprises metadata mapping records and corresponding parent-child node information.
5. The method for separately storing the metadata of the file system according to claim 4, wherein the allocating the mapping address of the user data according to whether the mapping node corresponding to the current user IO request is searched comprises:
and judging whether a corresponding mapping node is searched in the metadata mapping record tree or not, if so, releasing the metadata mapping record and distributing the user data mapping address after releasing the metadata mapping record tree node, if not, directly distributing the user data mapping address, and updating the user data mapping record file after distributing the user data mapping address.
6. The file system metadata split storage method according to 2~5, wherein said mapping record tree is a balanced binary tree indexed by virtual device disk offset, and wherein each node comprises virtual device disk offset, data length, physical memory offset, data mapping record number, parent node and child node information.
7. The file system metadata separation storage method of claim 1~5, wherein the intercepting of the metadata IO request and the user data IO request by the pre-registered virtual storage device further comprises starting the virtual storage device, reading physical device information from a configuration file, and loading data mapping records to the virtual storage device to respectively establish a metadata mapping record tree and a user data mapping record tree, wherein the metadata mapping record tree includes the metadata mapping records and corresponding parent and child node information, and the user data mapping record tree includes the user data mapping records and corresponding parent and child node information.
8. The file system metadata split storage method according to 1~5, wherein said virtual storage device is pre-registered on a generic block device layer, and a mapping driver of said virtual storage device has a function of loading a preset mapping rule from a specified location.
9. A file system metadata split storage apparatus, comprising:
the tag adding module is used for adding a designated tag for the metadata when the file system generates a metadata IO request;
and the virtual storage equipment is used for intercepting the metadata IO request and the user data IO request, identifying the type of the IO request according to whether the tag is attached or not when the IO request is intercepted, carrying out address mapping according to the type of the IO request and redirecting the request to the corresponding storage equipment, wherein if the request is the metadata IO request, the request is redirected to the metadata storage equipment, and if the request is the user data request, the request is redirected to the user data storage equipment.
10. A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed, implements the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211519205.1A CN115599704B (en) | 2022-11-30 | 2022-11-30 | File system metadata separate storage method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211519205.1A CN115599704B (en) | 2022-11-30 | 2022-11-30 | File system metadata separate storage method and device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115599704A true CN115599704A (en) | 2023-01-13 |
| CN115599704B CN115599704B (en) | 2023-03-17 |
Family
ID=84853244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211519205.1A Active CN115599704B (en) | 2022-11-30 | 2022-11-30 | File system metadata separate storage method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115599704B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070094269A1 (en) * | 2005-10-21 | 2007-04-26 | Mikesell Paul A | Systems and methods for distributed system scanning |
| CN104517067A (en) * | 2014-12-30 | 2015-04-15 | 华为技术有限公司 | Method, device and system for data access |
| CN104536903A (en) * | 2014-12-25 | 2015-04-22 | 华中科技大学 | Mixed storage method and system for conducting classified storage according to data attributes |
| US20170083549A1 (en) * | 2015-09-14 | 2017-03-23 | Emc Corporation | Tracing garbage collector for search trees under multi-version concurrency control |
| CN109299111A (en) * | 2018-11-14 | 2019-02-01 | 郑州云海信息技术有限公司 | A kind of metadata query method, apparatus, equipment and computer readable storage medium |
| CN109359062A (en) * | 2018-11-02 | 2019-02-19 | 郑州云海信息技术有限公司 | A kind of metadata read buffer method, device and equipment |
| CN109992708A (en) * | 2019-04-12 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus of metadata query, equipment and storage medium |
| CN111125447A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Metadata access method, device and equipment and readable storage medium |
| CN113946291A (en) * | 2021-10-20 | 2022-01-18 | 重庆紫光华山智安科技有限公司 | Data access method, device, storage node and readable storage medium |
-
2022
- 2022-11-30 CN CN202211519205.1A patent/CN115599704B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070094269A1 (en) * | 2005-10-21 | 2007-04-26 | Mikesell Paul A | Systems and methods for distributed system scanning |
| CN104536903A (en) * | 2014-12-25 | 2015-04-22 | 华中科技大学 | Mixed storage method and system for conducting classified storage according to data attributes |
| CN104517067A (en) * | 2014-12-30 | 2015-04-15 | 华为技术有限公司 | Method, device and system for data access |
| US20170083549A1 (en) * | 2015-09-14 | 2017-03-23 | Emc Corporation | Tracing garbage collector for search trees under multi-version concurrency control |
| CN109359062A (en) * | 2018-11-02 | 2019-02-19 | 郑州云海信息技术有限公司 | A kind of metadata read buffer method, device and equipment |
| CN109299111A (en) * | 2018-11-14 | 2019-02-01 | 郑州云海信息技术有限公司 | A kind of metadata query method, apparatus, equipment and computer readable storage medium |
| CN109992708A (en) * | 2019-04-12 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus of metadata query, equipment and storage medium |
| CN111125447A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Metadata access method, device and equipment and readable storage medium |
| CN113946291A (en) * | 2021-10-20 | 2022-01-18 | 重庆紫光华山智安科技有限公司 | Data access method, device, storage node and readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 张震伟等: "基于超元数据缓存的高速数据访问系统", 《广东通信技术》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115599704B (en) | 2023-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11068455B2 (en) | Mapper tree with super leaf nodes | |
| JP6767115B2 (en) | Safety device for volume operation | |
| EP3814928B1 (en) | System and method for early removal of tombstone records in database | |
| CN110799960A (en) | System and method for database tenant migration | |
| CN109445685B (en) | User mode file system processing method | |
| CN103399823B (en) | The storage means of business datum, equipment and system | |
| US11157299B2 (en) | Thin provisioning virtual desktop infrastructure virtual machines in cloud environments without thin clone support | |
| US20170160940A1 (en) | Data processing method and apparatus of solid state disk | |
| CN112912868A (en) | System and method for bulk removal of records in a database | |
| WO2023045407A1 (en) | I/o request processing method and apparatus, device and readable storage medium | |
| US8239427B2 (en) | Disk layout method for object-based storage devices | |
| CN111007990B (en) | Positioning method for quickly positioning data block references in snapshot system | |
| US11074012B2 (en) | Storage device, information processing system, and non-transitory computer-readable storage medium for storing program | |
| US7509473B2 (en) | Segmented storage system mapping | |
| CN111309262A (en) | Distributed storage cache reading and writing method | |
| CN115599704B (en) | File system metadata separate storage method and device and storage medium | |
| US11080299B2 (en) | Methods and apparatus to partition a database | |
| JP2005108239A (en) | Storage system for storing data in hierarchical data structure | |
| US20230409235A1 (en) | File system improvements for zoned storage device operations | |
| US6842843B1 (en) | Digital data storage subsystem including arrangement for increasing cache memory addressability | |
| US10235373B2 (en) | Hash-based file system | |
| US20230266919A1 (en) | Hint-based fast data operations with replication in object-based storage | |
| US20240119029A1 (en) | Data processing method and related apparatus | |
| CN106777288A (en) | A kind of quick fractionation virtualizes the system and method for disk mirroring | |
| CN116185949A (en) | Cache storage method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |