CN109189480B - File system starting method and device - Google Patents
File system starting method and device Download PDFInfo
- Publication number
- CN109189480B CN109189480B CN201810709288.8A CN201810709288A CN109189480B CN 109189480 B CN109189480 B CN 109189480B CN 201810709288 A CN201810709288 A CN 201810709288A CN 109189480 B CN109189480 B CN 109189480B
- Authority
- CN
- China
- Prior art keywords
- file system
- distributed file
- information
- version information
- metadata information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/4401—Bootstrapping
- G06F9/4411—Configuring for operating with peripheral devices; Loading of device drivers
Abstract
The application provides a file system starting method and a device, which are applied to a distributed file system, and the method comprises the following steps: creating a mirror image file for storing metadata information of the distributed file system on a disk of the distributed file system; updating metadata information stored in a mirror image file according to metadata in a memory when the metadata information in the memory of the distributed file system changes; and when the distributed file system is started, loading metadata information from the mirror image file for starting the distributed file system. Through the design, the time used in the starting process of the distributed file system can be shortened, and the starting speed is accelerated.
Description
Technical Field
The present application relates to the field of distributed system technologies, and in particular, to a method and an apparatus for starting a file system.
Background
In a distributed file system, Metadata (Metadata) information is used to describe attribute information such as the distributed location, file size, access rights, etc. of files in the distributed file system. All metadata information needs to be loaded first, and then the distributed file system can be started successfully, so that services are provided for the outside. At present, metadata information is usually loaded from a disk of a distributed file system for system startup, and the metadata information in the disk is usually updated correspondingly according to data information written into the disk.
However, in the starting process of the existing distributed file system, the time for loading the metadata information is long, and the user experience is poor.
Disclosure of Invention
In view of the above, the present application provides a file system booting method and apparatus to at least partially improve the above problem.
In a first aspect, an embodiment of the present application provides a file system starting method, which is applied to a distributed file system, and the method includes:
creating a mirror image file for storing metadata information in a disk of the distributed file system;
updating the metadata information stored in the image file according to the metadata in the memory when the metadata information in the memory of the distributed file system changes;
and when the distributed file is started, loading metadata information from the mirror image file for starting the distributed file system.
Optionally, according to the file system startup method provided in the first aspect of the embodiment of the present application, the metadata information includes version information, the distributed file system further includes a log file, which is used to cache write operation data for the distributed file system, and each write operation data carries version information of the write operation data; the method comprises the following steps:
after loading metadata information from the mirror image file, acquiring version information included in current metadata information in a memory of the distributed file system as first version information, and acquiring version information carried in write operation data recorded latest in the log file as second version information;
judging whether the first version information and the second version information are the same;
when the first version information is different from the second version information, updating write operation data carrying target version information in the log file to a disk of the distributed file system, and then loading metadata information including the target version information in the disk of the distributed file system to a memory of the distributed file system, wherein the target version information is version information which is newer than the first version information.
Optionally, according to the file system starting method provided in the first aspect of the embodiment of the present application, when the distributed file system is started, before the step of loading metadata information from the image file for starting the distributed file system, the method further includes:
judging whether the mirror image file is available;
when the mirror image file is available, loading metadata information from the mirror image file for starting the distributed file system;
and when the mirror image file is unavailable, updating the write operation data in the log file into a disk of the distributed file system, and loading metadata information from the disk of the distributed file system for starting the distributed file system.
Optionally, according to the file system starting method provided in the first aspect of the embodiment of the present application, the method further includes:
and when the distributed file system is successfully started or when an instruction for reading data information is received, updating the write operation data in the log file into a disk of the distributed file system.
Optionally, according to the file system starting method provided in the first aspect of the embodiment of the present application, a target thread independent from an I/O thread is preconfigured in the distributed file system; and when the metadata information in the memory of the distributed file system changes, the step of updating the metadata information stored in the image file according to the metadata information in the memory is executed in the target thread.
In a second aspect, an embodiment of the present application further provides a file system starting apparatus, which is applied to a distributed file system, where the apparatus includes:
the system comprises a mirror image file creating module, a metadata information storing module and a metadata information storing module, wherein the mirror image file creating module is used for creating a mirror image file for storing metadata information in a disk of the distributed file system;
the metadata storage module is used for updating the metadata information stored in the mirror image file according to the metadata in the memory when the metadata information in the memory of the distributed file system changes;
and the starting module is used for loading metadata information from the mirror image file when the distributed file is started, and is used for starting the distributed file system.
Optionally, according to the file system startup device provided in the second aspect of the embodiment of the present application, the metadata information includes version information, the distributed file system further includes a log file, which is used to cache write operation data for the distributed file system, and each write operation data carries version information of the write operation data; the device further comprises:
the version information acquisition module is used for acquiring version information included in the current metadata information in the memory of the distributed file system as first version information and acquiring version information carried in write operation data recorded latest in the log file as second version information after the start module loads the metadata information from the mirror image file;
the first judging module is used for judging whether the first version information is the same as the second version information; when the first judgment module determines that the first version information is different from the second version information, storing write operation data carrying target version information in the log file into a disk of the distributed file system, and then loading metadata information including the target version information in the disk of the distributed file system into a memory of the distributed file system, wherein the target version information is version information which is newer than the first version information.
Optionally, according to a second aspect of an embodiment of the present application, the file system starting apparatus further includes:
the second judgment module is used for loading metadata information from the mirror image file when the distributed file system is started, and judging whether the mirror image file is available before the distributed file system is started; when the mirror image file is available, triggering the starting module to load metadata information from the mirror image file for starting the distributed file system; and when the mirror image file is unavailable, storing the write operation data in the log file into a disk of the distributed file system, and loading metadata information from the disk of the distributed file system for starting the distributed file system.
Optionally, according to a second aspect of an embodiment of the present application, the file system starting apparatus further includes:
and the data loading module is used for storing the write operation data in the log file into a disk of the distributed file system when the distributed file system is successfully started or when an instruction for reading data information is received.
Optionally, according to the file system startup device provided in the second aspect of the embodiment of the present application, a target thread independent from an I/O thread is preconfigured in the distributed file system, and the metadata saving module runs in the target thread.
In a third aspect, the present application also provides a machine-readable storage medium having stored thereon computer instructions that, when executed, cause a processor to: the file system starting method provided by the first aspect of the embodiment of the application.
Compared with the prior art, the method has the following beneficial effects:
the file system starting method and device provided by the embodiment of the application are applied to a distributed file system. Creating a mirror image file for storing metadata information in a disk of the distributed file system, and updating the metadata information in the mirror image file according to the metadata information in a memory of the distributed file system when the metadata information in the memory changes. And when the distributed file system is started, loading metadata information from the mirror image file for starting the distributed file system. Therefore, the starting time of the distributed file system can be shortened, and the starting speed of the distributed file system is accelerated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
FIG. 1 is a diagram illustrating an architecture of a distributed file system in the prior art;
fig. 2 is a schematic structural diagram of a distributed file system according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a file system booting method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a file system booting method according to an embodiment of the present application;
fig. 5 is a functional block diagram of a file system boot apparatus according to an embodiment of the present disclosure.
Icon: 100. 200-a distributed file system; 110. 210-a magnetic disk; 211-mirror image file; 120. 220-log file; 230-a memory; 300-file system startup means; 310-image file creation module; 320-metadata saving module; 330-start module; 340-version information obtaining module; 350-a first judgment module; 360-a second judgment module; 370-a data loading module; 400-client side.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In practical application, data written into the distributed file system needs to be persistently stored in a disk of the distributed file system, when data is written, metadata information of the distributed file system changes, and correspondingly, the changed metadata information is also written into the disk. When the distributed file system is started, all metadata information is loaded from a disk and used for starting the distributed file system.
Because the head needs to swing to address each time the data is written on the disk of the distributed file system, and the write operation for writing the small file data into the distributed file system is more, if the small file data is written each time, the small file data is firstly addressed by the head swing and then written next time, the write data performance of the distributed file system is very low, therefore, in some embodiments, for the write operation for writing the small file data, the small file data is usually written into the log file first, and then the data in the log file is stored on the disk persistently. Here, the small file data generally refers to a file having a file data size within 1 MB.
In detail, as shown in fig. 1, a user typically accesses the distributed file system 100 through the client 400, when the distributed file system 100 receives a write operation request sent by the client 400 for writing small file data, the small file data is written into the log file 120, and when the small file data is successfully written into the log file 120, a message of successful writing is immediately returned to the client 400, so that the client 400 can immediately perform a next write operation. In the subsequent process, the data information in the log file 120 is persistently stored on the disk 110, and the metadata information on the disk 110 is updated correspondingly.
However, if the distributed file system 100 is restarted due to a power failure, a fault, and the like in a case that the data information in the log file 120 is not stored on the disk 110, in the restart process, the distributed file system 100 needs to store the data information in the log file 120 on the disk 110 persistently, update the metadata information on the disk 110 based on the data information, load the metadata information from the disk 110, and construct cluster topology information based on the loaded metadata information, so as to implement the start of the distributed file system 100. The process of storing all of the data information in the log file 120 to the disk 110 is time consuming, which results in a long startup time of the distributed file system 100 and a poor user experience.
The defects existing in the above solutions are the results obtained after the inventor has practiced and studied carefully, so the discovery process of the above problems and the solutions proposed by the following embodiments of the present application for the above problems should be the contribution of the inventor to the present application in the process of invention.
Based on the above research, embodiments of the present application provide a file system booting method and apparatus applied to a distributed file system, so as to accelerate the booting speed of the distributed file system, which will be described in detail below.
As shown in fig. 2, which is a schematic view of an architecture of a distributed file system 200 according to an embodiment of the present disclosure, the distributed file system 200 includes a disk 210, a log file 220, a memory 230, and an image file 211 stored on the disk 210, where the image file 211 is a file created in advance for storing metadata information of the distributed file system 200. In this embodiment of the present application, the write operation data in the log file 220 may be specifically stored in a Key-Value pair (Key-Value) form, and fig. 2 is a schematic diagram of the log file 220 being stored in the Key-Value form. In addition, the memory 230 may be a memory 230 of a server or a host deployed with a metadata service, and the image file 211 may be on a disk 210 of the server or the host deployed with the data service, or may be on a disk 210 of the server or the host deployed with the metadata service.
It should be understood that, in the present embodiment, the data service and the metadata service may also be deployed on the same server or host, and the present embodiment is not limited thereto.
Fig. 3 is a schematic flowchart of a file system booting method provided in an embodiment of the present application, where the file booting method is applied to the distributed file system 200 shown in fig. 2, and the method is described in detail below with reference to fig. 3.
In step S301, an image file 211 for storing metadata information of the distributed file system 200 is created on the disk 210 of the distributed file system 200.
The image file 211 is the pre-created image file 211 shown in fig. 2, and the image file 211 is persistently stored on the disk 210 of the distributed file system 200, and data on the image file 211 is not lost in the case of system power failure.
In step S302, when the metadata information in the memory 230 of the distributed file system 200 changes, the metadata information stored in the image file 211 is updated according to the metadata information in the memory 230.
In this embodiment, after the image file 211 is created, all the current metadata information in the memory 230 of the distributed file system 200 is saved in the image file 211. After that, by means of incremental updating, when it is detected that the metadata information in the memory 230 of the distributed file system 200 has changed, for example, an operation such as addition, deletion, or modification occurs, the metadata information stored in the image file 211 is updated according to the changed metadata information.
It should be appreciated that in the present embodiment, step S302 is typically performed during the operation of the distributed file system 200.
Optionally, in this embodiment, step S302 may be implemented by taking a snapshot. In detail, when the image file 211 is created, a snapshot is made of all current metadata information of the distributed file system 200, and the obtained metadata snapshot is stored in the image file 211. After that, once it is detected that the metadata information in the memory 230 of the distributed file system 200 changes, a delta snapshot is made on the metadata information in the memory 230, and the delta snapshot is stored in the image file 211.
Optionally, in this embodiment, a target thread independent from the I/O thread is created in the distributed file system in advance, and step S302 may be executed in the target thread.
Through the design, the process of updating the metadata information increment into the image file 211 is not in an I/O process, is independently operated, and does not block I/O operation aiming at the distributed file system 200.
Step S303, when the distributed file system 200 is started, loading metadata information from the image file 211, for starting the distributed file system 200.
In this embodiment, when the distributed file system 200 is started, the metadata information is directly loaded from the image file 211 to the memory 230 of the distributed file system 200, so as to start the distributed file system 200 and provide external services based on the metadata information.
Therefore, the time for storing all the data information in the log file 220 to the disk 210 is saved in the starting process, the starting speed is accelerated, and the user experience is improved.
It should be noted that, in practical applications, when write operation data is written into the log file 220, the metadata information of the memory 230 is updated to a state corresponding to the write operation data, and in the embodiment of the present application, since the metadata information in the memory 230 is persistently stored in the image file 211 in real time, the metadata information finally loaded from the image file 211 to the memory 230 is usually the latest and most complete metadata information.
Optionally, in this embodiment, for each write operation, the distributed file system 200 may allocate version information to the write operation data, where the version information may be a transaction number (also referred to as "transaction ID"), and the write operation data corresponding to the write operation includes the version information, where the write operation data corresponding to any write operation may be an integral of the write operation instruction and the data to be written. Correspondingly, after the write operation data is written into the distributed file system 200, the metadata information of the distributed file system 200 changes, and the version information included in the changed metadata information is the same as the version information in the write operation data.
In order to further ensure that the metadata information finally loaded in the memory 230 of the distributed file system 200 is the latest and most complete metadata information, the file system starting method provided in this embodiment may further include four steps, i.e., step S401 to step S404, as shown in fig. 4.
Step S401, after loading the metadata information from the image file 211, obtains version information included in the current metadata information in the memory 230 of the distributed file system 200 as first version information, and obtains version information carried in the write operation data recorded latest in the log file 220 as second version information.
Step S402, determining whether the first version information and the second version information are the same.
In practical applications, there may be a case where all metadata information in the image file 211 cannot be loaded into the memory 230, or a small amount of metadata information is missing in the image file 211 due to a delay problem. After the metadata information is loaded from the image file 211 into the memory 230 of the distributed file system 200, the version information (i.e., the first version information) included in the metadata information in the memory 230 is the latest version of metadata information currently loaded in the memory 230. Since all the write operation data that is not updated to the disk 210 in time is recorded in the log file 220, whether the metadata information in the memory 230 of the distributed file system 200 has been loaded to the latest state can be determined by comparing whether the version information (i.e., the second version information) of the write operation data that is recorded in the log file 220 latest is the same as the first version information, and if the version information is the same as the first version information, it indicates that the metadata information has been loaded to the latest state, and the loading can be stopped. If not, it indicates that the metadata information in the memory 230 has not been loaded to the latest state, and at this time, the metadata information in the memory 230 can be loaded to the latest state through step 403.
Step S403, when the first version information is different from the second version information, updating the write operation data carrying the target version information in the log file 220 to the disk 210 of the distributed file system 200, and then loading the metadata information including the target version information in the disk 210 of the distributed file system 200 to the memory 230 of the distributed file system 200, where the target version information is version information that is newer than the first version information.
In this embodiment, taking the version information as the transaction number information as an example, the transaction numbers allocated to the write operations by the distributed file system 200 are sequentially increased according to the sequence of the write operations, that is, the transaction numbers allocated to the write operations are larger as the sequence of the write operations is later. In this case, the version information newer than the first version information may be version information larger than the first version information.
In step S403, since a part of the metadata information is already loaded from the image file 211, only the write operation data corresponding to the metadata information that is not updated needs to be persistently stored in the log file 220 on the disk 210 of the distributed file system 200, that is, the write operation data with the version information that is newer than the first version information carried in the log file 220 is persistently stored in the disk 210. In this way, the distributed file system 200 correspondingly changes the metadata information based on the write operation data written to the disk 210, and then loads the metadata information, which includes version information that is newer than the first version information, from the disk 210 into the memory 230 of the distributed file system 200, thereby ensuring that the metadata information in the memory 230 of the distributed file system 200 is loaded to the latest state.
Through the above design, during the startup process of the distributed file system 200, only a small portion of the data information in the log file 220 needs to be stored on the disks 210 of the distributed file system 200, which is much shorter than the time taken to store all the data information in the log file 220 on the disks 210 of the distributed file system 200.
Optionally, in practical application, in order to avoid a situation that the system cannot be started due to damage of the image file 211, the file system starting method provided in the embodiment of the present application may further include the following steps:
when the distributed file system 200 is started, judging whether the mirror image file 211 is available;
when the image file 211 is available, a step of loading metadata information from the image file 211 for starting the distributed file system 200 is performed;
when the image file 211 is not available, the write operation data in the log file 220 is updated to the disk 210 of the distributed file system 200, and then the metadata information is loaded from the disk 210 of the distributed file system 200, so as to start the distributed file system 200.
Whether the image file 211 is available or not can be judged by whether the data in the image file 211 can be read or not, if the data can be read from the image file 211, the image file 211 is judged to be available, otherwise, the image file 211 is judged to be unavailable.
Optionally, in this embodiment, the file system starting method may further include the following steps:
when the distributed file system 200 is successfully started or when an instruction for reading data information is received, the write operation data in the log file 220 is updated to the disks 210 of the distributed file system 200.
Through the design, the two processes of loading the metadata information and loading the data information are separated, and decoupling between the two processes is realized, that is, the metadata information in the memory 230 is persistently stored in the mirror image file 211 in real time, and the data information in the log file 220 is not required to be persistently stored in the disk 210 in real time, so that the data information in the log file 220 is prevented from being loaded before the metadata information is loaded, and the time for starting the distributed file system 200 is saved.
Fig. 5 is a functional block diagram of a file system boot apparatus 300 according to an embodiment of the present disclosure, which is applied to the distributed file system 200 shown in fig. 2. The file system boot apparatus 300 includes an image file 211 creation module, a metadata saving module 320, and a boot module 330.
The image file 211 creating module is configured to create an image file 211 in the disk 210 of the distributed file system 200, where the image file 211 is used to store metadata information of the distributed file system 200.
In this embodiment, the description of the image file 211 creation module may specifically refer to the detailed description of step S301 shown in fig. 3, that is, step S301 may be performed by the image file 211 creation module.
The metadata saving module 320 is configured to update the metadata information stored in the image file 211 according to the metadata information in the memory 230 when the metadata information in the memory 230 of the distributed file system 200 changes.
In the present embodiment, the description about the metadata saving module 320 may specifically refer to the description about step S302 shown in fig. 3, that is, step S302 may be performed by the metadata saving module 320.
Optionally, in this embodiment, a target thread independent from the I/O thread is created in the distributed file system in advance, and the metadata saving module 320 runs in the target thread.
Optionally, in this embodiment, the metadata information may include version information, and the log file 220 is configured to cache write operation data for the distributed file system 200, where each write operation data carries the version information of the write operation data. The metadata can be newly changed after the write operation data is written, and the version information included in the changed metadata information is the same as the version information carried by the write operation data. Alternatively, the version information may be a transaction number.
In order to ensure that the metadata information in the memory 230 is loaded to the latest state, the file system booting apparatus 300 according to this embodiment may further include a version information obtaining module 340 and a first determining module 350.
The version information obtaining module 340 is configured to, after the start module 330 loads the metadata information from the image file 211, obtain version information included in current metadata information in the memory 230 of the distributed file system 200 as first version information, and obtain version information carried in write operation data that is newly recorded in the log file 220 as second version information.
In the present embodiment, the description about the version information acquisition module 340 may refer to the detailed description of step S401 shown in fig. 4, that is, step S401 may be performed by the version information acquisition module 340.
The first determining module 350 is configured to determine whether the first version information and the second version information are the same; when the first determining module 350 determines that the first version information is different from the second version information, the write operation data carrying the target version information in the log file 220 is stored in the disk 210 of the distributed file system 200, and then the metadata information including the target version information in the disk 210 of the distributed file system 200 is loaded in the memory 230 of the distributed file system 200, wherein the target version information is version information that is newer than the first version information.
In the present embodiment, the description about the first determining module 350 may refer to the detailed description of step S402 and step S403 shown in fig. 4, that is, step S402 and step S403 may be executed by the first determining module 350.
Optionally, in this embodiment, the file system starting apparatus 300 may further include a second determining module 360.
The second determining module 360 is configured to load metadata information from the image file 211 when the distributed file system 200 is started, and determine whether the image file 211 is available before starting the distributed file system 200; when the image file 211 is available, the start module 330 is triggered again to load metadata information from the image file 211, so as to start the distributed file system 200; when the image file 211 is not available, the write operation data in the log file 220 is stored in the disk 210 of the distributed file system 200, and then the metadata information is loaded from the disk 210 of the distributed file system 200 for starting the distributed file system 200.
Optionally, in this embodiment, the file system starting apparatus 300 may further include a data loading module 370.
The data loading module 370 is configured to store the write operation data in the log file 220 in the disk 210 of the distributed file system 200 when the distributed file system 200 is successfully started or when an instruction for reading data information is received.
For the above modules, reference may be made to the detailed description of the relevant steps in the above description, and details are not described here.
To sum up, the embodiments of the present application provide a method and an apparatus for starting a file system, where an image file for storing metadata information is created in a disk of a distributed file system, and when metadata information in a memory of the distributed file system changes, the metadata information in the image file is updated according to the metadata information in the memory. And when the distributed file system is started, loading metadata information from the mirror image file for starting the distributed file system. Therefore, the starting time of the distributed file system can be shortened, and the starting speed of the distributed file system is accelerated.
The foregoing description has been given for illustrative purposes only and is not intended to be limiting, and numerous modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (8)
1. A file system starting method is applied to a distributed file system, and is characterized by comprising the following steps:
creating a mirror image file for storing metadata information of the distributed file system on a disk of the distributed file system;
updating the metadata information stored in the image file according to the metadata information in the memory when the metadata information in the memory of the distributed file system changes;
when the distributed file is started, loading metadata information from the mirror image file for starting the distributed file system;
the distributed file system further comprises a log file for caching write operation data aiming at the distributed file system, wherein each write operation data carries the version information of the write operation data; the method comprises the following steps:
after loading metadata information from the mirror image file, acquiring version information included in current metadata information in a memory of the distributed file system as first version information, and acquiring version information carried in write operation data recorded latest in the log file as second version information;
judging whether the first version information and the second version information are the same;
when the first version information is different from the second version information, updating write operation data carrying target version information in the log file to a disk of the distributed file system, and then loading metadata information including the target version information in the disk of the distributed file system to a memory of the distributed file system, wherein the target version information is version information which is newer than the first version information.
2. The file system booting method of claim 1 wherein at boot time of the distributed file system, metadata information is loaded from the image file for use before the step of booting the distributed file system, the method further comprising:
judging whether the mirror image file is available;
when the mirror image file is available, loading metadata information from the mirror image file for starting the distributed file system;
and when the mirror image file is unavailable, updating the write operation data in the log file into a disk of the distributed file system, and loading metadata information from the disk of the distributed file system for starting the distributed file system.
3. The file system startup method according to claim 1 or 2, characterized in that the method further comprises:
and when the distributed file system is successfully started or when an instruction for reading data information is received, updating the write operation data in the log file into a disk of the distributed file system.
4. The file system starting method according to any one of claims 1-2, wherein a target thread independent of an I/O thread is created in advance in the distributed file system; and when the metadata information in the memory of the distributed file system changes, the step of updating the metadata information stored in the image file according to the metadata information in the memory is executed in the target thread.
5. A file system startup device applied to a distributed file system, the device comprising:
the system comprises a mirror image file creating module, a metadata information storing module and a metadata information storing module, wherein the mirror image file creating module is used for creating a mirror image file for storing metadata information of the distributed file system in a disk of the distributed file system;
the metadata storage module is used for updating the metadata information stored in the mirror image file according to the metadata information in the memory when the metadata information in the memory of the distributed file system changes;
the starting module is used for loading metadata information from the mirror image file when the distributed file is started and is used for starting the distributed file system;
the distributed file system further comprises a log file for caching write operation data aiming at the distributed file system, wherein each write operation data carries the version information of the write operation data; the device further comprises:
the version information acquisition module is used for acquiring version information included in the current metadata information in the memory of the distributed file system as first version information and acquiring version information carried in write operation data recorded latest in the log file as second version information after the start module loads the metadata information from the mirror image file;
the first judging module is used for judging whether the first version information is the same as the second version information; when the first judgment module determines that the first version information is different from the second version information, storing write operation data carrying target version information in the log file into a disk of the distributed file system, and then loading metadata information including the target version information in the disk of the distributed file system into a memory of the distributed file system, wherein the target version information is version information which is newer than the first version information.
6. The file system startup device of claim 5, characterized in that the device further comprises:
the second judgment module is used for loading metadata information from the mirror image file when the distributed file system is started, and judging whether the mirror image file is available before the distributed file system is started; when the mirror image file is available, triggering the starting module to load metadata information from the mirror image file for starting the distributed file system; and when the mirror image file is unavailable, storing the write operation data in the log file into a disk of the distributed file system, and loading metadata information from the disk of the distributed file system for starting the distributed file system.
7. The file system startup device according to claim 5 or 6, characterized in that the device further comprises:
and the data loading module is used for storing the write operation data in the log file into a disk of the distributed file system when the distributed file system is successfully started or when an instruction for reading data information is received.
8. The file system starting device according to any one of claims 5 to 6, wherein a target thread independent of an I/O thread is created in advance in the distributed file system, and the metadata saving module runs in the target thread.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810709288.8A CN109189480B (en) | 2018-07-02 | 2018-07-02 | File system starting method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810709288.8A CN109189480B (en) | 2018-07-02 | 2018-07-02 | File system starting method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109189480A CN109189480A (en) | 2019-01-11 |
| CN109189480B true CN109189480B (en) | 2021-11-09 |
Family
ID=64948780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810709288.8A Active CN109189480B (en) | 2018-07-02 | 2018-07-02 | File system starting method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109189480B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109996262B (en) * | 2019-03-19 | 2022-04-22 | 新华三技术有限公司 | AC starting method and device |
| CN110795404B (en) * | 2019-10-31 | 2023-04-07 | 京东方科技集团股份有限公司 | Hadoop distributed file system and operation method and repair method thereof |
| CN114706622B (en) * | 2022-03-10 | 2023-08-18 | 北京百度网讯科技有限公司 | Method, device, equipment, medium and product for starting model service |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789004A (en) * | 2010-01-22 | 2010-07-28 | 北京北大众志微系统科技有限责任公司 | Rapid-start method and device of Flash file system |
| CN102722536A (en) * | 2012-05-22 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for realizing snapshot based on distributed file system |
| CN103761275A (en) * | 2014-01-09 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Management method for metadata in distributed file system |
| CN104933133A (en) * | 2015-06-12 | 2015-09-23 | 中国科学院计算技术研究所 | Metadata snap storage and access method in distributed file system |
| US9336219B2 (en) * | 2014-03-03 | 2016-05-10 | Netapp, Inc. | Distributed file system snapshot |
| CN107357688A (en) * | 2017-07-28 | 2017-11-17 | 广东神马搜索科技有限公司 | Distributed system and its fault recovery method and device |
| CN107480150A (en) * | 2016-06-07 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of file loading method and device |
| CN107506356A (en) * | 2016-06-14 | 2017-12-22 | 北京京东尚科信息技术有限公司 | Data processing method and its system |
| US10268719B2 (en) * | 2015-06-23 | 2019-04-23 | International Business Machines Corporation | Granular buffering of metadata changes for journaling file systems |
-
2018
- 2018-07-02 CN CN201810709288.8A patent/CN109189480B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789004A (en) * | 2010-01-22 | 2010-07-28 | 北京北大众志微系统科技有限责任公司 | Rapid-start method and device of Flash file system |
| CN102722536A (en) * | 2012-05-22 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for realizing snapshot based on distributed file system |
| CN103761275A (en) * | 2014-01-09 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Management method for metadata in distributed file system |
| US9336219B2 (en) * | 2014-03-03 | 2016-05-10 | Netapp, Inc. | Distributed file system snapshot |
| CN104933133A (en) * | 2015-06-12 | 2015-09-23 | 中国科学院计算技术研究所 | Metadata snap storage and access method in distributed file system |
| US10268719B2 (en) * | 2015-06-23 | 2019-04-23 | International Business Machines Corporation | Granular buffering of metadata changes for journaling file systems |
| CN107480150A (en) * | 2016-06-07 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of file loading method and device |
| CN107506356A (en) * | 2016-06-14 | 2017-12-22 | 北京京东尚科信息技术有限公司 | Data processing method and its system |
| CN107357688A (en) * | 2017-07-28 | 2017-11-17 | 广东神马搜索科技有限公司 | Distributed system and its fault recovery method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109189480A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107870968B (en) | Performing real-time updates to a file system volume | |
| EP2329360B1 (en) | Managing cache data and metadata | |
| US8135914B2 (en) | Managing cache data and metadata | |
| US8694984B2 (en) | Memory apparatus and method of updating firmware of the memory apparatus | |
| US9081639B2 (en) | System and method for remotely re-imaging a computer system | |
| CN109189480B (en) | File system starting method and device | |
| US20060224639A1 (en) | Backup system, program and backup method | |
| US20230393840A1 (en) | File update method and apparatus, device and storage medium | |
| US7818557B2 (en) | Method for re-imaging a computer system | |
| US20200250038A1 (en) | Facilitating the identification of a service operating system when a main operating system fails | |
| US9715398B2 (en) | Program code loading method of application and computing system using the same | |
| CN111399760A (en) | NAS cluster metadata processing method and device, NAS gateway and medium | |
| JP2015114750A (en) | Examination program, information processing device, and information processing method | |
| US10969970B2 (en) | Storage optimization of database in volatile and non-volatile storing unit | |
| CN114443166B (en) | Shared file processing method, device and medium | |
| US11226875B2 (en) | System halt event recovery | |
| CN114880293A (en) | Software starting acceleration method and device and computing equipment | |
| US20130262804A1 (en) | Data duplication system, data duplication method, and program thereof | |
| US20220413733A1 (en) | mon service migration method, apparatus, and device, and readable storage medium | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN113190244A (en) | Method and device for upgrading wireless module, computer equipment and storage medium | |
| US20230342215A1 (en) | Method to optimize storage partition rediscovery | |
| JP7355876B2 (en) | Program startup method, equipment, and storage medium | |
| CN117215835A (en) | Equipment starting method and device, electronic equipment and storage medium | |
| CN116244034A (en) | File lock realization method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |