CN117591345B - Ceph-specific data synchronization method, device, equipment and storage medium - Google Patents

Ceph-specific data synchronization method, device, equipment and storage medium Download PDF

Info

Publication number
CN117591345B
CN117591345B CN202410072385.6A CN202410072385A CN117591345B CN 117591345 B CN117591345 B CN 117591345B CN 202410072385 A CN202410072385 A CN 202410072385A CN 117591345 B CN117591345 B CN 117591345B
Authority
CN
China
Prior art keywords
data
area
disk
bitmap
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202410072385.6A
Other languages
Chinese (zh)
Other versions
CN117591345A (en
Inventor
郭厥论
聂小峰
许宇峰
黄华宾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Baike Data Technology Shenzhen Co ltd
Original Assignee
Baike Data Technology Shenzhen Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Baike Data Technology Shenzhen Co ltd filed Critical Baike Data Technology Shenzhen Co ltd
Priority to CN202410072385.6A priority Critical patent/CN117591345B/en
Publication of CN117591345A publication Critical patent/CN117591345A/en
Application granted granted Critical
Publication of CN117591345B publication Critical patent/CN117591345B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1456—Hardware arrangements for backup
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1415—Saving, restoring, recovering or retrying at system level
    • G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1415—Saving, restoring, recovering or retrying at system level
    • G06F11/1438—Restarting or rejuvenating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Human Computer Interaction (AREA)
  • Library & Information Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data synchronization method, a device, equipment and a storage medium for Ceph, wherein the method comprises the steps of reading a disk bitmap stored in a file layer when a system is powered off and restarted; determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time; and reading the target data stored in the area to be synchronized and resynchronizing the target data. The application builds the disk bitmap of the disk for backing up the data in advance, records the storage position of the data in the disk through the disk bitmap, can determine the region to be synchronized which is likely to send the data to be lost according to the disk bitmap when the power-off restarting occurs, then analyzes the region to be synchronized to determine the target data and resynchronizes the target data, thus the time required by the data analysis can be reduced, and the data backup efficiency can be improved.

Description

Ceph-specific data synchronization method, device, equipment and storage medium
Technical Field
The present application relates to the field of data backup technologies, and in particular, to a method, an apparatus, a device, and a storage medium for synchronization of data with respect to Ceph.
Background
The data backup is the basis of disaster recovery, and the process of data backup is to synchronize the data of the production end to the target end. However, during the data backup process, there may be a case where data is lost due to a power burst abnormality. The current method for coping with such a situation is to perform full-disc analysis on the target terminal when an abnormal power-off situation occurs, so as to determine missing data generated by the abnormality. Although this method can avoid data loss, it takes a lot of time to analyze the data, which affects the data backup efficiency.
There is thus a need for improvements and improvements in the art.
Disclosure of Invention
The application aims to solve the technical problem of providing a data synchronization method, a device, equipment and a storage medium for Ceph aiming at the defects of the prior art.
In order to solve the above technical problem, a first aspect of the present application provides a data synchronization method for Ceph, the method comprising:
when the system is powered off and restarted, reading a disk bitmap stored in a file layer;
Determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time;
And reading the target data stored in the area to be synchronized and resynchronizing the target data.
In the data synchronization method for Ceph, the generating time and the region position of the disk generating data change are recorded in the disk bitmap, and determining the region to be synchronized corresponding to the disk according to the disk bitmap and the power-off restarting time specifically includes:
Reading the power-off restarting time;
Traversing each generation time in the disk bitmap to obtain a target time closest to the power-off restarting moment, and determining a target area position corresponding to the disk according to the target time;
and taking the magnetic disk area corresponding to the target area position as an area to be synchronized.
The method for synchronizing data for Ceph, wherein determining, according to the target time, a target area position corresponding to the magnetic disk specifically includes:
Matching the target time with the power-off restarting time;
when the target time is matched with the power-off restarting time, determining a target area position corresponding to the magnetic disk according to the target time;
and when the target time is not matched with the power-off restarting time, taking all the area positions of the magnetic disk as target area positions.
The data synchronization method for Ceph, wherein the method further comprises the following steps:
Creating an empty file in a file layer in advance, and taking the empty file as a bitmap file;
dividing the magnetic disk into a plurality of storage areas with area positions, and generating a magnetic disk bitmap according to the plurality of storage areas with the area positions, wherein when the storage areas generate data changes, the magnetic disk bitmap records the generation time of the data changes generated in the storage areas;
the disk bitmap is stored in the bitmap file so as to be read from the file layer to the disk bitmap by reading the bitmap file.
The data synchronization method for Ceph, wherein the method further comprises the following steps:
when the magnetic disk generates data change, recording the position and the generation time of the area generating the data change, and adding the position of the area into log information;
and acquiring the position of the area generating the data change and the generation time through the log information, and writing the generation time into the disk bitmap.
The data synchronization method for Ceph, wherein writing the generation time into the disk bitmap specifically includes:
The generation time is written into the disk bitmap through a fast device layer.
The method for synchronizing data for Ceph, wherein the reading the target data stored in the to-be-synchronized area and resynchronizing the target data specifically includes:
reading target data stored in a region to be synchronized, and obtaining original data corresponding to the target data;
and determining difference data between the target data and the original data, and synchronizing the difference data.
The second aspect of the present application provides a data synchronization device for Ceph, where the data synchronization device for Ceph specifically includes:
the reading module is used for reading the disk bitmap stored in the file layer when the system is powered off and restarted;
the determining module is used for determining a region to be synchronized corresponding to the disk according to the disk bitmap and the power-off restarting time;
And the synchronization module is used for reading the target data stored in the area to be synchronized and resynchronizing the target data.
A third aspect of the application provides a computer readable storage medium storing one or more programs executable by one or more processors to implement steps in a data synchronization method for Ceph as described in any one of the above.
A fourth aspect of the present application provides a server, including: a processor and a memory;
the memory has stored thereon a computer readable program executable by the processor;
the processor, when executing the computer-readable program, implements the steps in a data synchronization method for Ceph as described in any one of the above.
The beneficial effects are that: compared with the prior art, the application provides a data synchronization method, a device, equipment and a storage medium for Ceph, wherein the method comprises the steps of reading a disk bitmap stored in a file layer when a system is powered off and restarted; determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time; and reading the target data stored in the area to be synchronized and resynchronizing the target data. The application builds the disk bitmap of the disk for backing up the data in advance, records the storage position of the data in the disk through the disk bitmap, can determine the region to be synchronized which is likely to send the data to be lost according to the disk bitmap when the power-off restarting occurs, then analyzes the region to be synchronized to determine the target data and resynchronizes the target data, thus the time required by the data analysis can be reduced, and the data backup efficiency can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a data synchronization method for Ceph according to an embodiment of the present application.
Fig. 2 is a flow chart illustrating a process of constructing a disk bitmap of the data synchronization method for Ceph according to an embodiment of the present application.
Fig. 3 is a schematic structural diagram of a data synchronization device for Ceph according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a server provided in an embodiment of the present application.
Detailed Description
The embodiment of the application provides a data synchronization method, device, equipment and storage medium for Ceph, and the application is further described in detail below with reference to the accompanying drawings and the embodiment for making the purposes, technical schemes and effects of the application clearer and more definite. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
It will be understood by those skilled in the art that all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
It should be understood that the sequence number and the size of each step in this embodiment do not mean the sequence of execution, and the execution sequence of each process is determined by the function and the internal logic of each process, and should not be construed as limiting the implementation process of the embodiment of the present application.
Through researches, the data backup is the basis of disaster recovery, and the process of data backup is to synchronize the data of the production end to the target end. However, during the data backup process, there may be a case where data is lost due to a power burst abnormality. The current method for coping with such a situation is to perform full-disc analysis on the target terminal when an abnormal power-off situation occurs, so as to determine missing data generated by the abnormality. Although this method can avoid data loss, it takes a lot of time to analyze the data, which affects the data backup efficiency.
In order to solve the above problems, in the embodiment of the present application, when the system is powered off and restarted, a disk bitmap stored in a file layer is read; determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time; and reading the target data stored in the area to be synchronized and resynchronizing the target data. The application builds the disk bitmap of the disk for backing up the data in advance, records the storage position of the data in the disk through the disk bitmap, can determine the region to be synchronized which is likely to send the data to be lost according to the disk bitmap when the power-off restarting occurs, then analyzes the region to be synchronized to determine the target data and resynchronizes the target data, thus the time required by the data analysis can be reduced, and the data backup efficiency can be improved.
The application will be further described by the description of embodiments with reference to the accompanying drawings.
The embodiment provides a data synchronization method for Ceph, as shown in FIG. 1, the method includes:
s10, when the system is powered off and restarted, the disk bitmap stored in the file layer is read.
Specifically, the disk bitmap corresponds to a disk, and is used to reflect the disk area in the disk where the data change occurs. That is, the disk bitmap is configured for the disk, and the disk area where the data change occurs in the disk can be determined by the disk bitmap. In the embodiment of the application, the disk bitmap records the generation time of the data change generated by the disk and the area position, wherein the area position refers to the storage position of the data in the disk, and the generation time refers to the last change time of the data change generated by the area position, so that the storage area of the last data received before the restarting is powered off by the disk bitmap.
Further, the disk bitmap can be generated when the disk is used as a backup data segment, and the disk bitmap can be updated in real time in the data backup process, so that the generation time and the region position of the disk generated data change recorded by the disk bitmap are matched with the actual stored data in the disk, and the accuracy of target data needing to be backed up again after power failure can be improved.
In some implementations, as shown in fig. 2, the data synchronization method for Ceph further includes:
Creating an empty file in a file layer in advance, and taking the empty file as a bitmap file;
dividing the magnetic disk into a plurality of storage areas with area positions, and generating a magnetic disk bitmap according to the plurality of storage areas with the area positions, wherein when the storage areas generate data changes, the magnetic disk bitmap records the generation time of the data changes generated in the storage areas;
the disk bitmap is stored in the bitmap file so as to be read from the file layer to the disk bitmap by reading the bitmap file.
Specifically, the bitmap file is used to store a disk bitmap. The creation operation of the empty file can be triggered based on the disk bitmap generation requirement, can be triggered based on preset time, and can be triggered based on user operation. In the embodiment of the application, the creation operation of the null file is triggered based on the disk bitmap generation requirement, namely, when the disk bitmap is required to be built for the disk, the creation operation of the bitmap file is triggered, and a null file is created based on the creation operation to serve as the bitmap file, and meanwhile, the bitmap file is associated with the disk, so that the bitmap file corresponding to the disk can be quickly determined.
In order to facilitate recording of data changes in a disk, the disk may be divided into storage areas, and an area position may be configured for each storage area, by which the storage area of data in the disk is reflected. For example, an area number is allocated for each storage area, and the area number of each storage area is used as an area position, or a start storage address and a stop storage address corresponding to each storage area are used as the area positions of the storage areas. In addition, it should be noted that, after the disk is divided into a plurality of storage areas, the generation time in the disk bitmap is the time when the data change occurs in the storage area corresponding to the area position.
Further, since the area to be synchronized needs to be determined according to the area position, when the power is turned off and restarted, there is a general need that the data received in a preset time period (for example, 1s,2s, etc.) before the power is turned off and restarted may be lost, so that when the area division is performed on the disk, the disk division basis (for example, 1.5 times, 1.8 times, or 2 times of the data amount) may be determined according to the data amount of the data that may be received in the preset time period. In one exemplary implementation, the disk is divided according to an amount of data that is greater than a predetermined length of time, for example, the predetermined length of time is 2s, the amount of data received by the predetermined length of time is 500M, and 1g is used as a division unit to divide the disk.
According to the method, the disk division basis is determined based on the data volume, so that one storage area or two storage areas can be directly selected as the areas to be synchronized when the power is turned off and restarted, the determination quantity of the areas to be synchronized can be improved, the data volume to be analyzed can be reduced as much as possible, the data backup efficiency can be improved, and the influence of the power off and restarting on the service is reduced.
When the disk bitmap is built for the disk, whether the disk stores the existing data or not can be checked, and when the disk does not store the existing data, the disk is directly divided and the disk bitmap is built; when the existing data is stored, reading the stored area corresponding to the existing data and the non-stored area except the existing storage area; then, dividing the existing storage area and the non-storage area respectively, recording the area position of each storage area for each storage area obtained by dividing the existing storage area, and taking the dividing moment as the generation time corresponding to the area position; and recording the area position of each storage area for each storage area obtained by dividing the non-storage area, and setting the generation time corresponding to the area position as null.
After the disk bitmap is created, the disk bitmap is stored as a bitmap file, and the disk bitmap is synchronously updated when the disk changes data. That is, when the disk generates data change, the generation time recorded in the disk bitmap by the disk area generating the data change is synchronously updated, so that the disk bitmap can accurately reflect the data change condition in the disk in real time.
In one implementation manner, the data synchronization method for Ceph further includes an update process of the disk bitmap, where the update process of the disk bitmap may be:
when the magnetic disk generates data change, recording the position and the generation time of the area generating the data change, and adding the position of the area into log information;
and acquiring the position of the area generating the data change and the generation time through the log information, and writing the generation time into the disk bitmap.
Specifically, the data change means that there is data written to the magnetic disk, that is, when data is written to the magnetic disk by a write operation, a storage area where the data is written and a write time are recorded, the area position of the storage area is taken as the area position where the data change is generated, and the write time is taken as the generation time. After recording the area position and the generation time, in order to update the disk bitmap based on the area position and the generation time, the area position and the generation time need to be updated into the disk bitmap located at the file layer.
Further, when the creation time is written into the disk bitmap, the creation time can be directly written into the disk bitmap by calling a file interface provided by the file layer, wherein the creation time is written into the disk bitmap, and the creation time is updated by adopting the existing creation time of the area position corresponding to the creation time in the disk bitmap. However, in practical use, the data backup process may be jammed, or even jammed for a long time, due to invoking the file interface of the file layer to perform the disk bitmap update. Therefore, in order to avoid the above problem, the embodiment of the present application is implemented by combining log information and a fast device layer when synchronizing the region position and the generation time to the disk bitmap. Specifically, the recorded generation time and the recorded region position are recorded in log information, the generation time and the recorded region position are transmitted to a fast device layer in a log information mode, the log information is read in the fast device layer to obtain the generation time and the recorded region position, and the generation time is sucked into a disk bitmap through the fast device layer. The application places the updating operation of the disk bitmap on the fast device layer for execution, thus avoiding the problem of file interface calling failure caused by that the file layer finishes the previous data processing and then takes the data as the arrival, and further avoiding the problem of blocking caused by calling the file interface to update the disk bitmap.
And S20, determining a region to be synchronized corresponding to the disk according to the disk bitmap and the power-off restarting time.
Specifically, the recording area position and the generation time in the disk bitmap, after the power-off restart time is acquired, the area position where the write operation is performed at the time of power-off can be determined from the disk bitmap, that is, the storage area where data loss is likely to occur can be determined.
Based on this, in one implementation manner, determining, according to the disk bitmap and the power-off restart time, the region to be synchronized corresponding to the disk specifically includes:
Reading the power-off restarting time;
Traversing each generation time in the disk bitmap to obtain a target time closest to the power-off restarting moment, and determining a target area position corresponding to the disk according to the target time;
and taking the magnetic disk area corresponding to the target area position as an area to be synchronized.
Specifically, traversing each generation time in the disk bitmap refers to comparing each generation time in the disk bitmap with a power-off restart time to obtain a target time nearest to the power-off restart time. After the target time is acquired, reading a target area position corresponding to the target time, and taking the target area position as a corresponding magnetic disk area as an area to be synchronized. Of course, in practical application, in order to improve accuracy of data synchronization after power failure, when the target time is acquired, the target time and the preamble time adjacent to the target time may be acquired, then a time interval between the target time and the preamble time is calculated, if the time interval is greater than a preset duration, the region position corresponding to the target time is taken as the target region position, and if the time interval is less than or equal to the preset duration, the region position corresponding to the target time and the region position corresponding to the preamble time are taken together as the target region position. The preamble time refers to the generation time which is the second closest to the power-off restart time.
In one implementation manner, the determining, according to the target time, the target area position corresponding to the magnetic disk specifically includes:
Matching the target time with the power-off restarting time;
when the target time is matched with the power-off restarting time, determining a target area position corresponding to the magnetic disk according to the target time;
and when the target time is not matched with the power-off restarting time, taking all the area positions of the magnetic disk as target area positions.
Specifically, since Ma Chongqi may be established after the power is off, the restart may be performed at intervals. Therefore, after the target time is obtained, the target time and the power-off restarting time can be matched, wherein the matching of the target time and the power-off restarting time means that the time interval between the target time and the power-off restarting time is compared with the preset interval time, and when the time interval is smaller than the preset interval time, the matching of the target time and the power-off restarting time is indicated, and then the target time can be taken as the time period of the preset duration before the termination time. Otherwise, when the time interval is greater than or equal to the preset interval time, the target time is not matched with the power-off restarting time, and then full disc analysis is needed, namely, all the area positions of the magnetic disc are used as target area positions.
S30, reading the target data stored in the area to be synchronized, and resynchronizing the target data.
Specifically, after the region to be synchronized is acquired, all data stored in the region to be synchronized may be used as target data, and the target data may be re-synchronized. In addition, in practical applications, the data in the region to be synchronized may be partially inconsistent with the original data, so that the region to be synchronized may be analyzed to determine the target data. Based on the above, the reading the target data stored in the to-be-synchronized area and resynchronizing the target data specifically includes: reading target data stored in a region to be synchronized, and obtaining original data corresponding to the target data; and determining difference data between the target data and the original data, and synchronizing the difference data. According to the embodiment of the application, the difference data can be determined and re-synchronized only by analyzing the target data, so that on one hand, the data loss caused by power failure and restarting can be avoided, and on the other hand, the synchronization efficiency of the data synchronization can be improved.
In summary, the present embodiment provides a data synchronization method for Ceph, which includes reading a disk bitmap stored in a file layer when a system is powered off and restarted; determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time; and reading the target data stored in the area to be synchronized and resynchronizing the target data. The application builds the disk bitmap of the disk for backing up the data in advance, records the storage position of the data in the disk through the disk bitmap, can determine the region to be synchronized which is likely to send the data to be lost according to the disk bitmap when the power-off restarting occurs, then analyzes the region to be synchronized to determine the target data and resynchronizes the target data, thus the time required by the data analysis can be reduced, and the data backup efficiency can be improved.
Based on the above data synchronization method for Ceph, this embodiment provides a data synchronization device for Ceph, as shown in fig. 3, where the data synchronization for Ceph specifically includes:
The reading module 100 is configured to read a disk bitmap stored in the file layer when the system is powered off and restarted;
the determining module 200 is configured to determine a region to be synchronized corresponding to the disk according to the disk bitmap and the power-off restart time;
And the synchronization module 300 is used for reading the target data stored in the area to be synchronized and resynchronizing the target data.
Based on the above-described data synchronization method for Ceph, the present embodiment provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps in the data synchronization method for Ceph as described in the above-described embodiments.
Based on the above data synchronization method for Ceph, the present application also provides a server, as shown in fig. 4, which includes at least one processor (processor) 20; a display screen 21; and a memory (memory) 22, which may also include a communication interface (Communications Interface) 23 and a bus 24. Wherein the processor 20, the display 21, the memory 22 and the communication interface 23 may communicate with each other via a bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may invoke logic instructions in the memory 22 to perform the methods of the embodiments described above.
Further, the logic instructions in the memory 22 described above may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product.
The memory 22, as a computer readable storage medium, may be configured to store a software program, a computer executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 performs functional applications and data processing, i.e. implements the methods of the embodiments described above, by running software programs, instructions or modules stored in the memory 22.
The memory 22 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the server, etc. In addition, the memory 22 may include high-speed random access memory, and may also include nonvolatile memory. For example, a plurality of media capable of storing program codes such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or a transitory storage medium may be used.
In addition, the specific processes that the storage medium and the plurality of instruction processors in the server side load and execute are described in detail in the above method, and are not stated here.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (7)

1. A method of data synchronization for Ceph, the method comprising:
when the system is powered off and restarted, reading a disk bitmap stored in a file layer;
Determining a region to be synchronized corresponding to a disk according to the disk bitmap and the power-off restarting time;
Reading target data stored in the area to be synchronized and resynchronizing the target data;
wherein the method further comprises:
when the disk generates data change, recording the area position and the generation time for generating the data change, adding the area position into log information, and transmitting the generation time and the area position to a fast device layer in a log information mode;
Acquiring the region position and the generation time for generating the data change through the log information, and writing the generation time into the disk bitmap through a fast device layer, wherein the generation time is written into the disk bitmap through the fast device layer, specifically, the log information is read at the fast device layer to obtain the generation time and the region position, and then the generation time is written into the disk bitmap through the fast device layer;
The method further comprises the steps of:
Creating an empty file in a file layer in advance, and taking the empty file as a bitmap file;
dividing the magnetic disk into a plurality of storage areas with area positions, and generating a magnetic disk bitmap according to the plurality of storage areas with the area positions, wherein when the storage areas generate data changes, the magnetic disk bitmap records the generation time of the data changes generated in the storage areas;
Storing the disk bitmap in the bitmap file so as to read the disk bitmap from the file layer by reading the bitmap file;
the method for dividing the magnetic disk into a plurality of storage areas carrying area positions comprises the following steps:
checking whether existing data are stored in a magnetic disk, and dividing the magnetic disk into a plurality of storage areas carrying area positions when the existing data are not stored; when the existing data is stored, reading an existing storage area corresponding to the existing data and an unrecorded area except the existing storage area, respectively carrying out area division on the existing storage area and the unrecorded area, recording the area position of each storage area for each storage area obtained by the division of the existing storage area, and taking the division time as the generation time corresponding to the area position; and recording the area position of each storage area for each storage area obtained by dividing the non-storage area, and setting the generation time corresponding to the area position as null.
2. The method for data synchronization for Ceph according to claim 1, wherein the disk bitmap records a generation time and a region position of a disk generated data change, and determining a region to be synchronized corresponding to a disk according to the disk bitmap and a power-off restart time specifically includes:
Reading the power-off restarting time;
Traversing each generation time in the disk bitmap to obtain a target time closest to the power-off restarting moment, and determining a target area position corresponding to the disk according to the target time;
and taking the magnetic disk area corresponding to the target area position as an area to be synchronized.
3. The method for synchronizing data for Ceph according to claim 2, wherein said determining a target area location corresponding to the disk according to the target time comprises:
Matching the target time with the power-off restarting time;
when the target time is matched with the power-off restarting time, determining a target area position corresponding to the magnetic disk according to the target time;
and when the target time is not matched with the power-off restarting time, taking all the area positions of the magnetic disk as target area positions.
4. The method for synchronizing data for Ceph according to claim 1, wherein said reading the target data stored in the region to be synchronized and resynchronizing the target data specifically comprises:
reading target data stored in a region to be synchronized, and obtaining original data corresponding to the target data;
and determining difference data between the target data and the original data, and synchronizing the difference data.
5. The data synchronization device for Ceph is characterized by specifically comprising the following steps:
the reading module is used for reading the disk bitmap stored in the file layer when the system is powered off and restarted;
the determining module is used for determining a region to be synchronized corresponding to the disk according to the disk bitmap and the power-off restarting time;
The synchronization module is used for reading the target data stored in the area to be synchronized and resynchronizing the target data;
Wherein, the data synchronization device for Ceph is further configured to:
when the disk generates data change, recording the area position and the generation time for generating the data change, adding the area position into log information, and transmitting the generation time and the area position to a fast device layer in a log information mode;
Acquiring the region position and the generation time for generating the data change through the log information, and writing the generation time into the disk bitmap through a fast device layer, wherein the generation time is written into the disk bitmap through the fast device layer, specifically, the log information is read at the fast device layer to obtain the generation time and the region position, and then the generation time is written into the disk bitmap through the fast device layer;
The data synchronization device for Ceph is further configured to:
Creating an empty file in a file layer in advance, and taking the empty file as a bitmap file;
dividing the magnetic disk into a plurality of storage areas with area positions, and generating a magnetic disk bitmap according to the plurality of storage areas with the area positions, wherein when the storage areas generate data changes, the magnetic disk bitmap records the generation time of the data changes generated in the storage areas;
Storing the disk bitmap in the bitmap file so as to read the disk bitmap from the file layer by reading the bitmap file;
the method for dividing the magnetic disk into a plurality of storage areas carrying area positions comprises the following steps:
checking whether existing data are stored in a magnetic disk, and dividing the magnetic disk into a plurality of storage areas carrying area positions when the existing data are not stored; when the existing data is stored, reading an existing storage area corresponding to the existing data and an unrecorded area except the existing storage area, respectively carrying out area division on the existing storage area and the unrecorded area, recording the area position of each storage area for each storage area obtained by the division of the existing storage area, and taking the division time as the generation time corresponding to the area position; and recording the area position of each storage area for each storage area obtained by dividing the non-storage area, and setting the generation time corresponding to the area position as null.
6. A computer readable storage medium storing one or more programs executable by one or more processors to implement the steps in the data synchronization method for Ceph as claimed in any one of claims 1-4.
7. A server, comprising: a processor and a memory;
the memory has stored thereon a computer readable program executable by the processor;
the processor, when executing the computer readable program, implements the steps in the data synchronization method for Ceph according to any one of claims 1-4.
CN202410072385.6A 2024-01-18 2024-01-18 Ceph-specific data synchronization method, device, equipment and storage medium Active CN117591345B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410072385.6A CN117591345B (en) 2024-01-18 2024-01-18 Ceph-specific data synchronization method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410072385.6A CN117591345B (en) 2024-01-18 2024-01-18 Ceph-specific data synchronization method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN117591345A CN117591345A (en) 2024-02-23
CN117591345B true CN117591345B (en) 2024-05-14

Family

ID=89911905

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410072385.6A Active CN117591345B (en) 2024-01-18 2024-01-18 Ceph-specific data synchronization method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117591345B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118035003A (en) * 2024-04-10 2024-05-14 深圳市科力锐科技有限公司 Data synchronization method, device and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104407940A (en) * 2014-11-26 2015-03-11 上海爱数软件有限公司 Method for quickly recovering CDP system
CN105824689A (en) * 2016-03-30 2016-08-03 国云科技股份有限公司 Storage synchronization method for virtual machine
CN108334277A (en) * 2017-05-10 2018-07-27 中兴通讯股份有限公司 A kind of daily record write-in and synchronous method, device, system, computer storage media
CN110456984A (en) * 2019-06-21 2019-11-15 南京壹进制信息科技有限公司 The method that the block device of a kind of pair of Ceph storage carries out continuous data protection
CN112256492A (en) * 2020-12-22 2021-01-22 深圳市科力锐科技有限公司 Block device backup method, device and storage medium
CN115168307A (en) * 2022-07-22 2022-10-11 上海致景信息科技有限公司 Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104407940A (en) * 2014-11-26 2015-03-11 上海爱数软件有限公司 Method for quickly recovering CDP system
CN105824689A (en) * 2016-03-30 2016-08-03 国云科技股份有限公司 Storage synchronization method for virtual machine
CN108334277A (en) * 2017-05-10 2018-07-27 中兴通讯股份有限公司 A kind of daily record write-in and synchronous method, device, system, computer storage media
CN110456984A (en) * 2019-06-21 2019-11-15 南京壹进制信息科技有限公司 The method that the block device of a kind of pair of Ceph storage carries out continuous data protection
CN112256492A (en) * 2020-12-22 2021-01-22 深圳市科力锐科技有限公司 Block device backup method, device and storage medium
CN115168307A (en) * 2022-07-22 2022-10-11 上海致景信息科技有限公司 Data synchronization method, system, equipment and storage medium supporting breakpoint continuous transmission

Also Published As

Publication number Publication date
CN117591345A (en) 2024-02-23

Similar Documents

Publication Publication Date Title
CN117591345B (en) Ceph-specific data synchronization method, device, equipment and storage medium
US9367598B2 (en) Merging an out of synchronization indicator and a change recording indicator in response to a failure in consistency group formation
EP0671686B1 (en) Synchronous remote data duplexing
EP0672985B1 (en) Asynchronous remote data duplexing
US5504861A (en) Remote data duplexing
US10365978B1 (en) Synchronization of snapshots in a distributed consistency group
CN108319617B (en) Method and device for determining master-slave difference of database and switching control method and device
WO2016091069A1 (en) Data operation method and device
US10990312B2 (en) Method, apparatus, device and storage medium for processing data location of storage device
CN105824846B (en) Data migration method and device
CN110532123B (en) Fault transfer method and device of HBase system
CN109614054B (en) data reading method and system
CN111339118A (en) Kubernetes-based resource change history recording method and device
EP4300314A2 (en) Data backup method, apparatus and system
CN111930788B (en) Processing method, device and equipment of operation request, readable storage medium and system
WO2021082925A1 (en) Transaction processing method and apparatus
CN111930569A (en) System and method for backup and recovery of whole machine
CN111930707B (en) Method and system for correcting drive letter of windows cloud migration
EP2135166B1 (en) Service workload identification in a data storage system
US20070118605A1 (en) Method and computer system for information notification
CN117130871B (en) Parallel playback method and device for database logs and nonvolatile storage medium
CN114327289B (en) Data synchronization method and device and electronic equipment
CN117492935B (en) Mirror image data synchronization method, device, terminal and storage medium
CN117666950A (en) Hard disk data migration method, device, medium and NAS equipment
CN113971168A (en) Data synchronization method, device and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant