CN109947600B - File recovery method, device, equipment and medium based on Ceph - Google Patents
File recovery method, device, equipment and medium based on Ceph Download PDFInfo
- Publication number
- CN109947600B CN109947600B CN201910244724.3A CN201910244724A CN109947600B CN 109947600 B CN109947600 B CN 109947600B CN 201910244724 A CN201910244724 A CN 201910244724A CN 109947600 B CN109947600 B CN 109947600B
- Authority
- CN
- China
- Prior art keywords
- file
- file system
- target data
- offset
- intra
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a file recovery method, a device, equipment and a medium based on Ceph, wherein the method comprises the following steps: when the file system is abnormal, the target data file is extracted from RBD block equipment when the access to the target data file through the file system fails; the file system is established based on RBD block equipment, and the target data file is stored in the RBD block equipment; and restoring the target data file to the file system. According to the method, when the file system is abnormal, and the user cannot acquire the data file through the file system, the data file is recovered, and therefore the integral reliability of data reading and writing is ensured. In addition, the application also provides a file recovery device, equipment and medium based on the Ceph, and the beneficial effects are as described above.
Description
Technical Field
The present application relates to the field of data storage, and in particular, to a file recovery method, apparatus, device, and medium based on Ceph.
Background
Ceph (distributed file system) is a distributed storage system designed to provide high performance, reliability and scalability in data storage. The main objective of Ceph is to design a POSIX-based distributed file system without a single point of failure, so that data can be copied in a fault-tolerant and seamless manner, the Ceph needs to be constructed based on physical devices, and storage resources of the physical devices are packaged into RBD block devices in a Ceph framework, the RBD block devices are block storage adopted by the Ceph, and the Ceph provides RBD interfaces for users to call the RBD block devices and read and write data files on the RBD block devices.
When using the RBD block device, a user generally needs to package a file system on the basis of the RBD block device in advance. A file system is a method and data structure used by an operating system to specify files on a storage device or partition, to store and organize computer data, to access and to locate data via a fixed pattern. A partition or device needs to be formatted and the recording data structure written to disk before it can be used as a file system. Common file systems include ntfs, ext3, ext4, xfs, and so on. After a file system is established on the RBD block device, data files can be read and written from and to the corresponding RBD block device through the file system, but in the current process of reading and writing data through the file system, situations may occur in which a required data file cannot be obtained due to abnormality of the file system, such as damage to a data structure of the file system or mistaken deletion of a data file by a user, and it is difficult to ensure the overall reliability of data reading and writing.
Therefore, the file recovery method based on Ceph is provided to realize the recovery of the data file when the user cannot acquire the data file through the file system due to the abnormal file system, so as to ensure the overall reliability of data reading and writing, and is a problem to be solved by technical personnel in the field.
Disclosure of Invention
The application aims to provide a file recovery method, a file recovery device, equipment and a file recovery medium based on Ceph, so that when a user cannot acquire a data file through a file system due to the fact that the file system is abnormal, the data file can be recovered, and the overall reliability of data reading and writing is further ensured.
In order to solve the above technical problem, the present application provides a file recovery method based on Ceph, including:
when the file system is abnormal, the target data file is extracted from the RBD block equipment, and the access to the target data file through the file system is failed;
the file system is established based on RBD block equipment, and the target data file is stored in the RBD block equipment;
and restoring the target data file to the file system.
Preferably, when the file system exception causes a failure in accessing the target data file through the file system, the extracting the target data file in the RBD block device includes:
when the file system is abnormal, the access failure of the target data file through the file system is caused, and a pre-recorded intra-system offset interval of the target data file in the file system and a pre-recorded intra-block offset interval of the file system in the RBD block device are obtained;
and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
Preferably, the calculating a target offset interval of the target data file in the RBD block device according to the intra-block offset interval and the intra-system offset interval, and the extracting the target data file in the target offset interval in the RBD block device includes:
when the storage units between the intra-block offset interval and the intra-system offset interval are different, converting the storage units of the intra-block offset interval and the intra-system offset interval into the same target storage unit;
accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset;
accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset;
and extracting a target data file stored between the target starting position offset and the target end position offset in the RBD block device.
Preferably, the extracting the target data file stored between the target start position offset amount and the target end position offset amount in the RBD block device includes:
and extracting a target data file stored between the target starting position offset and the target final position offset in the RBD block device through a dd instruction or a rados instruction.
Preferably, when the file system exception causes a failure in accessing the target data file through the file system, the obtaining the intra-system offset interval of the pre-recorded target data file in the file system, and the intra-block offset interval of the pre-recorded file system in the RBD block device includes:
when the file system exception causes the access failure of the target data file through the file system and the file system is XFS, acquiring an intra-system offset interval in the file system by using an XFS _ bmap tool and acquiring a pre-recorded intra-block offset interval of the file system in RBD block equipment;
when the file system exception causes the access failure of the target data file through the file system and the file system is EXT, a stat instruction is utilized to acquire an intra-system offset interval in the file system and acquire a pre-recorded intra-block offset interval of the file system in the RBD block device.
Preferably, when the file system exception causes a failure in accessing the target data file through the file system, the obtaining the intra-system offset interval of the pre-recorded target data file in the file system, and the intra-block offset interval of the pre-recorded file system in the RBD block device includes:
when the file system exception causes the access failure of the target data file through the file system and the file system is XFS or EXT1 or EXT2, acquiring a pre-recorded intra-system offset interval of the target data file in the file system, and acquiring a pre-recorded intra-block offset interval in the file system by using a parked instruction;
when the file system exception causes the access failure of the target data file through the file system and the file system is EXT3 or EXT4, acquiring the intra-system offset interval of the pre-recorded target data file in the file system, and acquiring the pre-recorded intra-block offset interval in the file system by using the fdisk instruction.
Preferably, when the file system exception causes a failure in accessing the target data file through the file system, the obtaining the intra-system offset interval of the pre-recorded target data file in the file system, and the intra-block offset interval of the pre-recorded file system in the RBD block device includes:
when the file system exception causes the failure of accessing the target data file through the file system, acquiring an intra-system offset interval of the target data file in the file system, which is recorded in advance through a data dictionary format, and an intra-block offset interval of the file system in the RBD block device, which is recorded in advance through the data dictionary format.
In addition, this application still provides a file recovery device based on Ceph, includes:
the extraction module is used for extracting the target data file from the RBD block device when the file system is abnormal and the access to the target data file through the file system fails;
the file system is established based on RBD block equipment, and the target data file is stored in the RBD block equipment;
and the recovery module is used for recovering the target data file to the file system.
Preferably, the extraction module comprises:
the offset acquisition module is used for acquiring a pre-recorded intra-system offset interval of the target data file in the file system and a pre-recorded intra-block offset interval of the file system in the RBD block device when the file system is abnormal to cause the failure of accessing the target data file through the file system;
and the offset extraction module is used for calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
In addition, this application still provides a file recovery equipment based on Ceph, includes:
a memory for storing a computer program;
a processor for implementing the steps of the Ceph-based file recovery method as described above when executing the computer program.
Furthermore, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the Ceph-based file recovery method as described above.
According to the file recovery method based on the Ceph, when access to a target data file stored in RBD (radial basis function) block equipment fails through a file system, the file system is bypassed, the target data file is extracted from the RBD block equipment, and then the target data file is recovered to the file system. And the file system is established based on the RBD block device, and the target data file is stored in the RBD block device. Because the RBD block device has the characteristics of delayed deletion and overwriting, when a file system temporarily deletes a data file or the file system is damaged, the data file which has been landed in the RBD block device cannot be influenced, so that when the file system is abnormal and the access to the target data file through the file system fails, the target data file can still be retrieved in the RBD block device corresponding to the file system in a mode of bypassing the file system, and therefore when a user cannot acquire the data file through the file system due to the abnormal file system, the method realizes the recovery of the data file, and further ensures the integral reliability of data reading and writing. In addition, the application also provides a file recovery device, equipment and medium based on the Ceph, and the beneficial effects are as described above.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is a flow chart of a Ceph-based file recovery method disclosed herein;
FIG. 2 is a flowchart of a specific Ceph-based file recovery method disclosed herein;
FIG. 3 is a flowchart of a specific Ceph-based file recovery method disclosed herein;
FIG. 4 is a flowchart of a specific Ceph-based file recovery method disclosed herein;
FIG. 5 is a flowchart of a specific Ceph-based file recovery method disclosed herein;
FIG. 6 is a flowchart of a specific Ceph-based file recovery method disclosed herein;
FIG. 7 is a schematic structural diagram of a file recovery apparatus based on Ceph disclosed in the present application;
fig. 8 is a block diagram of a Ceph-based file recovery apparatus disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
After a file system is established on the RBD block device, data files can be read and written from and to the corresponding RBD block device through the file system, but in the current process of reading and writing data through the file system, situations may occur in which a required data file cannot be obtained due to abnormality of the file system, such as damage to a data structure of the file system or mistaken deletion of a data file by a user, and it is difficult to ensure the overall reliability of data reading and writing.
Therefore, the file recovery method based on the Ceph is provided, so that when a user cannot acquire the data file through the file system due to the fact that the file system is abnormal, the data file is recovered, and the overall reliability of data reading and writing is further ensured.
Referring to fig. 1, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S10: and when the file system exception causes the failure of accessing the target data file through the file system, extracting the target data file in the RBD block device.
The file system is built based on RBD block equipment, and the target data file is stored in the RBD block equipment.
It should be noted that the file system in this embodiment includes a file system in a Windows operating system environment and a file system in a Linux operating system environment, where the file system in the Windows operating system is subdivided by ntfs and the like, and the file system in the Linux operating system is subdivided by ext3, ext4, xfs and the like.
The file system exception in this step means that the data content in the file system is deleted by mistake
The RBD block device is a block device provided in the Ceph architecture and used for storing data files, and is essentially a block device generated in a logical form by dividing storage resources of physical devices on which the Ceph architecture is based, and a function of a 1T RBD block device is similar to that of a 1T SATA disk, and the mounted RBD block device can be used as a disk. When data in a file system on the upper layer of the RBD block device is deleted, as long as the covering and writing of the RBD block device are not triggered, the data are still kept unchanged in the RBD block device, so in the step, when the access to the target data file through the file system is failed due to the abnormality of the file system, namely the access to the target data file through the file system is failed due to the fault of the file system or the target data file in the file system is deleted by a user, the target data file can be obtained by directly extracting the data from the RBD block device in a mode of bypassing the file system.
Step S11: and restoring the target data file to the file system.
In the step, after the target data file is acquired in the RBD block device, the related information of the target data file is restored and recorded into the file system, so that the purpose of acquiring the target data file in the file system is achieved, and a user can smoothly access the target data file through the file system.
According to the file recovery method based on the Ceph, when the target data file stored in the RBD block device fails to be accessed through the file system, the file system is bypassed, the target data file is extracted from the RBD block device, and then the target data file is recovered to the file system. And the file system is established based on the RBD block device, and the target data file is stored in the RBD block device. Because the RBD block device has the characteristics of delayed deletion and overwriting, when a file system temporarily deletes a data file or the file system is damaged, the data file which has been landed in the RBD block device cannot be influenced, so that when the file system is abnormal and the access to the target data file through the file system fails, the target data file can still be retrieved in the RBD block device corresponding to the file system in a mode of bypassing the file system, and therefore when a user cannot acquire the data file through the file system due to the abnormal file system, the method realizes the recovery of the data file, and further ensures the integral reliability of data reading and writing.
Referring to fig. 2, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S20: when the file system exception causes the access failure of the target data file through the file system, the in-system offset interval of the pre-recorded target data file in the file system and the in-block offset interval of the pre-recorded file system in the RBD block device are obtained.
It should be noted that, because the file system is established based on the RBD block device, the data files that can be accessed in the file system are essentially stored based on the storage resources of the RBD block device, but because the data files need to be provided to the user through the file system, the file system needs to locate the corresponding data files according to the storage interval occupied by the data files in the file system, that is, the intra-system offset interval in this step. In addition, because the file system is established based on the RBD block device, and a plurality of file systems often can run on the RBD block device, and each file system occupies storage resources in a corresponding storage section in the RBD block device, a certain degree of address offset may exist between a starting storage address of the file system and a starting storage address of the RBD, that is, an intra-block offset section in this step.
In addition, the intra-block offset interval in this step may be pre-recorded when the file system is established based on the RBD block device; the intra-system offset interval in this step may be recorded in advance when writing a data file into the file system.
Step S21: and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
After the intra-system offset interval of the target data file in the file system and the intra-block offset interval of the file system in the RBD block device are obtained, the corresponding storage position of the target data file in the RBD block device, that is, the target offset interval in this step, can be located according to the intra-system offset interval and the intra-block offset interval, and then the target data file can be directly obtained in the RBD block device according to the target offset interval in a manner of bypassing the file system.
Step S22: and restoring the target data file to the file system.
In the embodiment, the specific offset interval of the target data file in the file system and the specific offset interval of the file system in the RBD block device are used for indirectly calculating to obtain the offset interval of the target data file in the RBD block device, so that the target data file is relatively accurately obtained in the RBD block device.
Referring to fig. 3, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S30: when the file system exception causes the access failure of the target data file through the file system, the in-system offset interval of the pre-recorded target data file in the file system and the in-block offset interval of the pre-recorded file system in the RBD block device are obtained.
Step S31: when the storage units between the intra-block offset section and the intra-system offset section are different, the storage unit of the intra-block offset section and the storage unit of the intra-system offset section are both converted into the same target storage unit.
It should be noted that, in the present embodiment, it is considered that there may be differences in storage units between storage addresses that are divided for a file system and an RBD block device, for example, for an XFS file system under the Linux operating system, the storage addresses may be divided in units of sectors (sectors); for the EXT4 file system under the Linux operating system, the division may be performed by a data block group with a fixed size; for the RBD block device, the storage address interval occupied by the file system established on the RBD block device may be divided by 16-ary numbers, so before calculating the target offset interval of the target data file in the RBD block device according to the intra-block offset interval and the intra-system offset interval, it is necessary to convert the storage units of the intra-block offset interval and the intra-system offset interval into the same target storage unit, where the target storage unit may be a preset certain storage unit, in order to ensure accurate calculation of the target offset interval of the target data file in the RBD block device.
Step S32: and accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset.
The starting position of the intra-block offset interval in the step refers to the original point position of the intra-block offset interval, and the target starting position offset of the target offset interval of the target data file in the RBD block device is obtained by further offsetting the starting position offset of the target data file in the file system based on the starting position offset of the file system in the RBD block device, so that the starting position offset of the intra-block offset interval in the step is accumulated based on the starting position offset of the intra-block offset interval, and the target starting position offset of the target data file in the target offset interval in the RBD block device is generated.
Step S33: and accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset.
The final position of the intra-block offset interval in this step refers to the end position of the intra-block offset interval relative to the origin, and since the target final position offset of the target offset interval of the target data file in the RBD block device is obtained by further offsetting the final position offset of the target data file in the file system based on the initial position offset of the file system in the RBD block device, the final position offset of the intra-block offset interval in this step is accumulated based on the final position offset of the intra-block offset interval, and the target final position offset of the target data file in the target offset interval in the RBD block device is generated.
In addition, the execution sequence between step S33 and step S32 in the present embodiment is not fixed, and may be executed at the same time.
Step S34: and extracting a target data file stored between the target starting position offset and the target final position offset in the RBD block device.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
In the previous step, the obtained target start position offset and the obtained target end position offset are the target offset interval of the target data file in the RBD block device, so that the target data file stored between the target start position offset and the target end position offset is extracted from the RBD block device in the present step.
Step S35: and restoring the target data file to the file system.
The embodiment further refines the calculation process of the target offset interval in which the target data file is stored in the RBD block device, and further ensures the overall accuracy of positioning the target data file in the RBD block device.
On the basis of the foregoing embodiment, the step of extracting the target data file stored between the target start position offset and the target end position offset in the RBD block device may specifically be:
and extracting a target data file stored between the target starting position offset and the target end position offset in the RBD block device through a dd instruction or a rados instruction.
Referring to fig. 4, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S40: and when the file system is XFS, acquiring an offset interval in the system in the file system by using an XFS _ bmap tool and recording the offset interval.
In order to ensure that the intra-system offset interval of the target data file in the file system can be normally acquired when the file system is XFS, an XFS _ bmap tool is used for acquiring the intra-system offset interval in the file system.
Step S41: and when the file system is EXT, acquiring and recording an offset interval in the system in the file system by using a stat instruction.
In order to ensure that the intra-system offset interval of the target data file in the file system can be normally acquired when the file system is of the EXT type, a stat instruction is used for acquiring the intra-system offset interval in the file system when the file system is of the EXT type.
Step S42: when the file system exception causes the access failure of the target data file through the file system, the in-system offset interval of the pre-recorded target data file in the file system and the in-block offset interval of the pre-recorded file system in the RBD block device are obtained.
Step S43: and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
Step S44: and restoring the target data file to the file system.
According to the method and the device, the matched instruction is selected to acquire the intra-system offset interval of the target data file in the file system according to the types of different file systems, and the reliability of the acquisition process of the intra-system offset interval is relatively ensured.
Referring to fig. 5, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S50: when the file system is XFS, EXT1 or EXT2, the intra-block offset interval is obtained in the file system by using the paired instruction and is recorded.
In order to ensure that the intra-block offset interval of the file system in the RBD block device can be smoothly acquired when the file system is XFS, EXT1 or EXT2, the intra-block offset interval is acquired in the file system by using a paired instruction.
Step S51: when the file system is EXT3 or EXT4, acquiring the intra-block offset interval in the file system by using the fdisk instruction and recording the intra-block offset interval.
In order to ensure that the intra-block offset section of the file system in the RBD block device can be smoothly acquired when the file system is EXT3 or EXT4, the intra-block offset section is acquired in the file system by using the fdisk instruction.
Step S52: when the file system exception causes the access failure of the target data file through the file system, the in-system offset interval of the pre-recorded target data file in the file system and the in-block offset interval of the pre-recorded file system in the RBD block device are obtained.
Step S53: and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
Step S54: and restoring the target data file to the file system.
According to the method and the device, the matched instruction is selected to acquire the intra-block offset interval of the file system in the RBD block equipment according to the types of different file systems, and the reliability of the acquisition process of the intra-block offset interval is relatively ensured.
Referring to fig. 6, an embodiment of the present application discloses a file recovery method based on Ceph, including:
step S60: when the file system exception causes the failure of accessing the target data file through the file system, acquiring an intra-system offset interval of the target data file in the file system, which is recorded in advance through a data dictionary format, and an intra-block offset interval of the file system in the RBD block device, which is recorded in advance through the data dictionary format.
It should be noted that the intra-system offset interval and the intra-block offset interval in this embodiment are recorded in the form of a data dictionary format, and the data dictionary format is essentially a format for recording a mapping relationship. The data dictionary uniquely records specific parameter values (Value) of each parameter (Key) in a Key-Value form, and the data dictionary can relatively accurately establish the corresponding relationship between the parameters and the parameter values, so in this embodiment, the offset interval of the target data file in the system of the file system and the offset interval of the file system in the block of the RBD block device are recorded in advance in a data dictionary format, and the accuracy of the user for acquiring the offset interval in the system and the offset interval in the block can be relatively ensured, thereby ensuring the reliability of the file recovery process.
Step S61: and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
Step S62: and restoring the target data file to the file system.
The application discloses an implementation mode under a specific application scene, which comprises the following steps:
the process of extracting the target data file mainly comprises three stages:
(1) Finding an intra-system offset interval of the file on the file system;
after the user file is stored in the file system, an offset value of the file system is occupied, and the offset value can be acquired through different file system tools.
For the Linux common file system XFS format, we can look using the XFS _ bmap tool:
root@zxw:～#xfs_bmap-lvp/home/zxw/file1
/home/zxw/file1:
EXT:FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0:[0..116503]:2056988232..2057104735 1(103478936..103595439)116504 00000
wherein 2056988232 and 2057104735 are used, and the number of sectors is 8 by knowing the starting and ending positions.
For the Linux file system EXT4 format, we can look at using the stat command:
root@aStor-EDS-2710:/home/zxw#stat file2
File:'file2'
Size:9Blocks:8IO Block:4096regular file
Device:801h/2049d Inode:655383Links:1
Access:(0644/-rw-r--r--)Uid:(0/root)Gid:(0/root)
Access:2019-01-10 16:59:22.738426084+0800
Modify:2019-01-10 16:59:22.738426084+0800
Change:2019-01-10 16:59:22.738426084+0800
655383, (655383-1)/8192 =80, which indicates that the inode is in the 80 th block group, (655383-1)% 8192=22, which indicates that the inode is in 22 entries of the inode table, and 80 × 32768=2621440, which indicates that the initial block position of the file inode in the 80 th block group is 621440.
(2) Finding an intra-block offset interval of the file system on an RBD block device of the Ceph;
after the RBD block device is formatted into the file system, a sector of the file system is 512B. The size of the RBD block device is fixed, and the size of a sector correspondingly occupied on a disk is fixed, so that the mapping relation of a sector of a file system corresponding to an object on the RBD block device needs to be found.
Using the paired command, as an example of XFS, the number of sectors can be found as 7814037168 sectors:
root@aStor-EDS-2710:～#parted-s/dev/sdt unit s print
Model:ATA WDC WD4002FYYZ-0(scsi)
Disk/dev/sdt:7814037168s
Sector size(logical/physical):512B/512B
Partition Table:loop
Disk Flags:
Number Start End Size File system Flags
1 0s 7814037167s 7814037168s xfs
the number of objects on the RBD block device can also be acquired through RBD info, and the number of objects is acquired by using the size/4M of the RBD block device.
Since each object is 4M, the method for mapping a single object to a sector is that the object id 4096 1024/512 is the mapping relationship between the object and the sector. Namely, it is
[0,8191] - - - - - - >0, and the 16-system conversion is 000000000000;
[8192,16383] - - - - >1, the conversion to 16-system is 000000000001.
(3) And (3) calculating the offset of the target data file on the ceph block device through the two offsets of the process (1) and the process (2), and extracting the object data.
Through the step (1), we can know that the start position of the target data file in the file system is 2056988232, and through the step (2), we can know that the start position of the partition position of the target data file system on the RBD block device is 0, so the offset position of the target data file is 20569888232 +0= 20569888232.
Thus, we have obtained the offset location of the target data file on the RBD block device, and then extract the data. There are two ways:
a, we can directly dd this rbd block device:
dd if＝/dev/rbd1of＝./testfile1bs＝512count＝8skip＝2056988232；
thus, the contents of the disk are directly read.
b, the second way is a direct rados get object:
knowing that the position of the target data file on the RBD block device is 20569888232 +0, 2057104735+0, namely 20569888232, 2057104735, the mapping relation corresponding to the object on the RBD block device can be obtained in step (2), and the mapping relation is converted into a 16-system object which is 000000000003D4D9,
the extraction method comprises the following steps:
rados-p ec_pool get 1000000000b.00000000testfile.
referring to fig. 7, an embodiment of the present application discloses a file recovery apparatus based on Ceph, including:
and the extraction module 10 is used for extracting the target data file from the RBD block device when the file system exception causes the failure of accessing the target data file through the file system.
The file system is established based on the RBD block device, and the target data file is stored in the RBD block device.
And a restoring module 11, configured to restore the target data file to the file system.
According to the file recovery device based on the Ceph, when the target data file stored in the RBD block device fails to be accessed through the file system, the file system is bypassed, the target data file is extracted from the RBD block device, and then the target data file is recovered to the file system. And the file system is established based on the RBD block device, and the target data file is stored in the RBD block device. Because the RBD block device has the characteristics of delayed deletion and overwriting, when a file system temporarily deletes a data file or the file system is damaged, the data file which is dropped in the RBD block device cannot be influenced, so that when the file system is abnormal and the access to the target data file through the file system fails, the target data file can still be retrieved in the RBD block device corresponding to the file system in a mode of bypassing the file system, and therefore when a user cannot acquire the data file through the file system due to the abnormality of the file system, the device realizes the recovery of the data file, and further ensures the integral reliability of data reading and writing.
On the basis of the foregoing embodiments, the embodiments of the present application further describe and optimize a file recovery apparatus based on Ceph. Specifically, the method comprises the following steps:
in one embodiment, the extraction module comprises:
the offset acquisition module is used for acquiring a pre-recorded intra-system offset interval of the target data file in the file system and a pre-recorded intra-block offset interval of the file system in the RBD block device when the file system is abnormal to cause that the access to the target data file through the file system fails;
and the offset extraction module is used for calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval and extracting the target data file in the target offset interval of the RBD block equipment.
In another specific embodiment, the offset extraction module includes:
and the unit conversion module is used for converting the storage unit of the intra-block offset interval and the storage unit of the intra-system offset interval into the same target storage unit when the storage units of the intra-block offset interval and the intra-system offset interval are different.
The initial offset calculation module is used for accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset;
the final position offset calculation module is used for accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset;
and the position extraction module is used for extracting a target data file stored between the target starting position offset and the target final position offset in the RBD block equipment.
In another embodiment, the location extraction module comprises:
and the instruction extracting module is used for extracting the target data file stored between the target starting position offset and the target end position offset in the RBD block equipment through a dd instruction or a rados instruction.
In another embodiment, the Ceph-based file recovery apparatus further includes:
and the first system offset acquisition module is used for acquiring the offset interval in the system in the file system by utilizing an XFS _ bmap tool when the file system is the XFS.
And the second system offset acquisition module is used for acquiring the intra-system offset interval in the file system by using a stat instruction when the file system is EXT.
In another specific embodiment, the Ceph-based file restoring apparatus further includes:
and the first intra-block offset acquisition module is used for acquiring the intra-block offset interval in the file system by using a paired instruction when the file system is XFS or EXT1 or EXT 2.
A second intra-block offset obtaining module, configured to, when the file system is EXT3 or EXT4, obtain the intra-block offset interval in the file system by using a fdisk instruction.
In another specific embodiment, the offset obtaining module includes:
and the dictionary offset acquisition module is used for acquiring an intra-system offset interval of the target data file in the file system recorded in advance through the data dictionary format and an intra-block offset interval of the file system in the RBD block device recorded in advance through the data dictionary format when the file system is abnormal to cause the failure of accessing the target data file through the file system.
Referring to fig. 8, an embodiment of the present application further discloses a file recovery device based on Ceph, including:
a memory 20 for storing a computer program;
a processor 21 for executing the computer program to implement the steps of:
when the file system is abnormal, the target data file is extracted from RBD block equipment when the access to the target data file through the file system fails; the file system is established based on RBD block equipment, and the target data file is stored in the RBD block equipment; and restoring the target data file to the file system.
According to the file recovery device based on the Ceph, when the target data file stored in the RBD block device fails to be accessed through the file system, the file system is bypassed, the target data file is extracted from the RBD block device, and then the target data file is recovered to the file system. And the file system is established based on the RBD block device, and the target data file is stored in the RBD block device. Because the RBD block device has the characteristics of delayed deletion and overwriting, when a file system temporarily deletes a data file or the file system is damaged, the data file which is dropped in the RBD block device cannot be influenced, so that when the file system is abnormal and the access to the target data file through the file system fails, the target data file can still be retrieved in the RBD block device corresponding to the file system in a mode of bypassing the file system, and therefore when the file system is abnormal and a user cannot acquire the data file through the file system, the device realizes the recovery of the data file, and further ensures the overall reliability of data reading and writing.
On the basis of the foregoing embodiments, the embodiments of the present application further describe and optimize a file recovery device based on Ceph. Specifically, the method comprises the following steps:
in this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: when the file system is abnormal, the access failure of the target data file through the file system is caused, and a system offset interval of the pre-recorded target data file in the file system and a block offset interval of the pre-recorded file system in the RBD block device are obtained; and calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block equipment.
In this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: when the storage units between the intra-block offset interval and the intra-system offset interval are different, converting the storage units of the intra-block offset interval and the intra-system offset interval into the same target storage unit; accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset; accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset; and extracting a target data file stored between the target starting position offset and the target final position offset in the RBD block device.
In this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: and extracting a target data file stored between the target starting position offset and the target final position offset in the RBD block device through a dd instruction or a rados instruction.
In this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: when the file system is XFS, acquiring an offset interval in the system in the file system by using an XFS _ bmap tool;
when the file system is EXT, obtaining an intra-system offset interval in the file system by using a stat instruction.
In this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: when the file system is XFS or EXT1 or EXT2, acquiring an intra-block offset interval in the file system by using a paired instruction; when the file system is EXT3 or EXT4, an intra-block offset interval is acquired in the file system by using the fdisk instruction.
In this embodiment, when the processor 21 executes the computer subprogram stored in the memory 20, the following steps may be specifically implemented: when the file system exception causes the failure of accessing the target data file through the file system, acquiring an intra-system offset interval of the target data file in the file system, which is recorded in advance through a data dictionary format, and an intra-block offset interval of the file system in the RBD block device, which is recorded in advance through the data dictionary format.
Further, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the Ceph-based file recovery method as described above. For the specific steps of the method, reference may be made to the web page data crawling method disclosed in the foregoing embodiment. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, which are not described herein again.
The computer-readable storage medium provided by the application, when access to a target data file stored in an RBD block device through a file system fails, bypasses the file system, extracts the target data file in the RBD block device, and then restores the target data file to the file system. The file system is established based on the RBD block device, and the target data file is stored in the RBD block device. Because the RBD block device has the characteristics of delayed deletion and overwriting, when a file system temporarily deletes a data file or the file system is damaged, the data file which is dropped in the RBD block device cannot be influenced, so that when the file system is abnormal and the access to the target data file through the file system fails, the target data file can still be retrieved in the RBD block device corresponding to the file system in a mode of bypassing the file system, therefore, when the file system is abnormal and a user cannot acquire the data file through the file system, the computer readable storage medium realizes the recovery of the data file, and further ensures the integral reliability of data reading and writing.
The method, the device, the equipment and the medium for file recovery based on Ceph provided by the application are described in detail above. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
Claims (8)
1. A file recovery method based on Ceph is characterized by comprising the following steps:
when the file system is abnormal, the target data file is extracted from RBD block equipment when the access to the target data file through the file system fails;
wherein the file system is built based on the RBD block device, and the target data file is stored in the RBD block device;
restoring the target data file to the file system;
when the file system exception causes the failure of accessing the target data file through the file system, the extracting the target data file in the RBD block device comprises the following steps:
when the file system exception causes the failure of accessing the target data file through the file system, acquiring a pre-recorded intra-system offset interval of the target data file in the file system and a pre-recorded intra-block offset interval of the file system in the RBD block device;
calculating a target offset interval of the target data file in the RBD block equipment according to the intra-block offset interval and the system intra-offset interval, and extracting the target data file in the target offset interval in the RBD block equipment;
the calculating a target offset interval of the target data file in the RBD block device according to the intra-block offset interval and the intra-system offset interval, and extracting the target data file in the target offset interval in the RBD block device includes:
converting the storage unit of the intra-block offset interval and the storage unit of the intra-system offset interval into the same target storage unit when the storage units of the intra-block offset interval and the intra-system offset interval are different;
accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset;
accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset;
and extracting the target data file stored between the target starting position offset and the target final position offset in the RBD block equipment.
2. The Ceph-based file recovery method of claim 1, wherein the extracting the target data file stored between the target start position offset and the target end position offset in the RBD block device comprises:
extracting the target data file stored between the target start position offset and the target end position offset in the RBD block apparatus by a dd instruction or a rados instruction.
3. The method for restoring a file according to claim 1, wherein the obtaining the pre-recorded intra-system offset interval of the target data file in the file system when the file system exception causes the access to the target data file via the file system to fail, and the pre-recorded intra-block offset interval of the file system in the RBD block device comprises:
when the file system exception causes the access failure of the target data file through the file system and the file system is XFS, acquiring the offset interval in the system in the file system by using an XFS _ bmap tool and acquiring the pre-recorded offset interval in the block of the file system in the RBD block device;
when the file system exception causes the failure of accessing the target data file through the file system and the file system is EXT, utilizing a stat instruction to acquire the intra-system offset interval in the file system and acquiring the pre-recorded intra-block offset interval of the file system in the RBD block device.
4. The method for restoring a file according to claim 1, wherein the obtaining the pre-recorded intra-system offset interval of the target data file in the file system when the file system exception causes the access to the target data file via the file system to fail, and the pre-recorded intra-block offset interval of the file system in the RBD block device comprises:
when the file system exception causes the access failure of the target data file through the file system and the file system is XFS, EXT1 or EXT2, acquiring a pre-recorded intra-system offset interval of the target data file in the file system, and acquiring the pre-recorded intra-block offset interval in the file system by using a parked instruction;
when the file system exception causes failure of access to the target data file through the file system and the file system is EXT3 or EXT4, acquiring a pre-recorded intra-system offset interval of the target data file in the file system, and acquiring the pre-recorded intra-block offset interval in the file system by using an fdisk instruction.
5. The method for restoring files according to any one of claims 2 to 4, wherein the obtaining of the pre-recorded intra-system offset interval of the target data file in the file system when the file system exception causes the access to the target data file via the file system to fail, and the pre-recorded intra-block offset interval of the file system in the RBD block device comprises:
when the file system exception causes failure of access to the target data file through the file system, acquiring a system offset interval of the target data file in the file system, which is recorded in advance through a data dictionary format, and an intra-block offset interval of the file system in the RBD block device, which is recorded in advance through the data dictionary format.
6. A file recovery apparatus based on Ceph, comprising:
the extraction module is used for extracting the target data file from the RBD block device when the file system is abnormal and the access to the target data file through the file system fails;
wherein the file system is built based on the RBD block device, and the target data file is stored in the RBD block device;
the recovery module is used for recovering the target data file to the file system;
the extraction module comprises:
an offset obtaining module, configured to, when the file system exception causes an access failure to the target data file through the file system, obtain a pre-recorded intra-system offset interval of the target data file in the file system and a pre-recorded intra-block offset interval of the file system in the RBD block device;
an offset extraction module, configured to calculate a target offset interval of the target data file in the RBD block device according to the intra-block offset interval and the intra-system offset interval, and extract the target data file in the target offset interval in the RBD block device;
the file recovery device based on the Ceph is further configured to:
converting the storage unit of the intra-block offset interval and the storage unit of the intra-system offset interval into the same target storage unit when the storage units of the intra-block offset interval and the intra-system offset interval are different;
accumulating the initial position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target initial position offset;
accumulating the final position offset of the offset interval in the system based on the initial position offset of the offset interval in the block to generate a target final position offset;
and extracting the target data file stored between the target starting position offset and the target final position offset in the RBD block equipment.
7. A file recovery device based on Ceph, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the Ceph-based file recovery method as claimed in any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of the Ceph-based file recovery method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910244724.3A CN109947600B (en) | 2019-03-28 | 2019-03-28 | File recovery method, device, equipment and medium based on Ceph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910244724.3A CN109947600B (en) | 2019-03-28 | 2019-03-28 | File recovery method, device, equipment and medium based on Ceph |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109947600A CN109947600A (en) | 2019-06-28 |
| CN109947600B true CN109947600B (en) | 2023-02-03 |
Family
ID=67012249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910244724.3A Active CN109947600B (en) | 2019-03-28 | 2019-03-28 | File recovery method, device, equipment and medium based on Ceph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109947600B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113778334A (en) * | 2021-08-30 | 2021-12-10 | 武汉市润普网络科技有限公司 | Distributed storage system based on electronic files |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1655089A (en) * | 2005-03-28 | 2005-08-17 | 北京紫光华宇软件股份有限公司 | Method for dumping data to mobile storage medium and extracting data therefrom |
| CN107135097A (en) * | 2017-04-14 | 2017-09-05 | 北京金融资产交易所有限公司 | The disaster tolerance system and disaster recovery method filed based on bookkeeping |
| CN109408291A (en) * | 2018-10-31 | 2019-03-01 | 郑州云海信息技术有限公司 | A kind of distributed memory system disk snapshot restoration methods, device and electronic equipment |
-
2019
- 2019-03-28 CN CN201910244724.3A patent/CN109947600B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1655089A (en) * | 2005-03-28 | 2005-08-17 | 北京紫光华宇软件股份有限公司 | Method for dumping data to mobile storage medium and extracting data therefrom |
| CN107135097A (en) * | 2017-04-14 | 2017-09-05 | 北京金融资产交易所有限公司 | The disaster tolerance system and disaster recovery method filed based on bookkeeping |
| CN109408291A (en) * | 2018-10-31 | 2019-03-01 | 郑州云海信息技术有限公司 | A kind of distributed memory system disk snapshot restoration methods, device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109947600A (en) | 2019-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8818950B2 (en) | Method and apparatus for localized protected imaging of a file system | |
| CN101937377B (en) | Data recovery method and device | |
| EP3159796B1 (en) | System and method for generating backups of a protected system from a recovery system | |
| KR100550288B1 (en) | Method for recovering data in ext2 file system, and computer-readable storage medium recorded with data-recover program | |
| CN103577121A (en) | High-reliability linear file access method based on nand flash | |
| CN102375695B (en) | A kind of access method of disk and computer system | |
| CN103577574A (en) | High-reliability linear file system based on nand flash | |
| CN112052121B (en) | Hard disk data recovery method and system | |
| CN101364218A (en) | Hard disk video recorder file systems and checking method thereof | |
| CN112115002B (en) | Method and device for recovering file from damaged or untrusted mechanical hard disk | |
| CN106709014B (en) | File system conversion method and device | |
| KR101593184B1 (en) | Method and apparatus for recovering partition based on file system metadata | |
| CN109947600B (en) | File recovery method, device, equipment and medium based on Ceph | |
| US20220342851A1 (en) | File system event monitoring using metadata snapshots | |
| CN112800007B (en) | Directory entry expansion method and system suitable for FAT32 file system | |
| CN103778032A (en) | Unix-like operating system backup method and device | |
| CN104794025A (en) | Method for quick verification on storage device | |
| TWI461904B (en) | Recovery method and device for linux using fat file system | |
| KR101670473B1 (en) | Method for recovering deleted data by mysql innodb database | |
| WO2009110665A1 (en) | Information storage medium for recording data according to journaling file system, and method of and apparatus for writing/recovering data using journaling file system | |
| CN107545022B (en) | Disk management method and device | |
| US10565057B2 (en) | Indirection-based storage system backups using markers including sets of serial numbers associated with segments | |
| CN114327292B (en) | File management method, system, electronic device and storage medium | |
| CN104991926A (en) | File system recovery method and system | |
| CN113590380A (en) | Database recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |