CN107797770B - Method and device for synchronizing disk state information - Google Patents
Method and device for synchronizing disk state information Download PDFInfo
- Publication number
- CN107797770B CN107797770B CN201711086203.7A CN201711086203A CN107797770B CN 107797770 B CN107797770 B CN 107797770B CN 201711086203 A CN201711086203 A CN 201711086203A CN 107797770 B CN107797770 B CN 107797770B
- Authority
- CN
- China
- Prior art keywords
- data block
- state
- bitmap
- synchronization
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a method and a device for synchronizing disk state information, wherein the method comprises the following steps: determining metadata of each data block in a memory at the current moment based on the operation type of the user data in each data block by the user in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every preset time length. The method and the device provided by the invention reduce the seek overhead and improve the data writing speed when maintaining the disk state information.
Description
Technical Field
The present invention relates to the field of computer storage, and in particular, to a method and an apparatus for synchronizing disk state information.
Background
The magnetic disk is the most commonly used auxiliary memory for storing information which is not needed to be used immediately at present, and once the information is needed, the magnetic disk exchanges data with the computer in batches, and the magnetic disk is a backup of a main memory, and has the advantages of large storage capacity, high reliability and low price. In order to effectively utilize the disk space and improve the access efficiency to the file, it is necessary to properly manage the disk resources.
The current common disk space management method is to store disk state information by using a bitmap (bitmap), i.e. one or more bits are used to represent a value corresponding to a certain key. Assuming we use a bit to represent a state, there are two possible values (0 or 1) for the state, Key is a number, and the location where the corresponding value is stored is just Key 1 bit. On the one hand, because bits are used to store data, a large amount of storage space can be saved, so in general, bitmaps are used to store a large number of simple states. On the other hand, because the data structure of the bitmap is a continuous storage structure, the bitmap saves space, and when the computer memory and the external memory are synchronized, the computer memory can be completely loaded/synchronized to the external memory only by one operation.
However, in some specific requirements, the modification of the state bitmap needs to be immediately saved in the external memory to execute the subsequent processes. On a mechanical disk, this can result in significant seek overhead, thereby reducing the operational efficiency of the entire disk.
For example, assume that a data block has three states: uninitialized, initialized, written data. A bitmap is used to save the state of all data blocks in the block of disk. When the state of the data block is changed to "written data", the state of the bitmap needs to be synchronized to the disk before the data is actually written. Otherwise, if unexpected power failure occurs at this time, the state of the data block is inconsistent with the state in the bitmap, so that the content of the bitmap is not credible. However, bitmaps are continuously stored in the disk, and the storage location is fixed, so that when data is written, if the content of the bitmaps needs to be updated every time of writing, the magnetic head is forced to frequently move back and forth between the bitmap storage location and the data writing location. This can incur a large seek overhead, thereby reducing the efficiency of the entire data write.
Therefore, it is an urgent technical problem to provide a method for synchronizing disk state information between a mechanical disk and a computer memory with low seek overhead and improved data writing efficiency.
Disclosure of Invention
The invention provides a method and a device for synchronizing disk state information, which aim to solve the technical problems of high seek overhead and low data writing efficiency in the existing disk state storage method.
In a first aspect, the present invention provides a method for synchronizing disk state information from a computer memory to a mechanical disk, where the space of the mechanical disk includes a data block space and a state bitmap space; the data block space is composed of a plurality of data block groups, each data block group is composed of a preset number of continuous data blocks, each data block stores metadata and user data, the disk state information includes a data block state, a data block group synchronization state and a data block synchronization state, and the data block state, the data block group synchronization state and the data block synchronization state are stored in the computer memory in a bitmap form, the method includes:
determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block;
if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space;
and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
In a second aspect, an embodiment of the present invention provides a method for loading disk state information from a mechanical disk to a computer memory, where the disk state information includes a data block state, a data block group synchronization state, and a data block synchronization state, and the data block state and the data block group synchronization state are stored in the mechanical disk in a bitmap form, where the method includes:
loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory;
and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
In a third aspect, the present invention provides a device for synchronizing disk state information from a computer memory to a mechanical disk, where the space of the mechanical disk includes a data block space and a state bitmap space; the data block space is composed of a plurality of data block groups, each data block group is composed of a preset number of continuous data blocks, each data block stores metadata and user data, the disk state information includes a data block state, a data block group synchronization state and a data block synchronization state, and the data block state, the data block group synchronization state and the data block synchronization state are stored in the computer memory in a bitmap form, the apparatus includes:
the determining unit is used for determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block;
if the judging unit judges that the user data of the data block at the current moment is in an intermediate state or a final state, the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, and the data block group synchronization state is updated to the state bitmap space;
and the synchronization unit synchronizes the data block state bitmap of the data block in the memory to the state bitmap space every preset time length.
In a fourth aspect, an embodiment of the present invention provides an apparatus for loading disk state information from a mechanical disk into a computer memory, where the disk state information includes a data block state, a data block group synchronization state, and a data block synchronization state, and the data block state, the data block group synchronization state, and the data block synchronization state are stored in the mechanical disk in a bitmap form, and the apparatus includes:
the loading unit loads the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory;
and the creating unit is used for creating the data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
In a fifth aspect, the present invention provides an electronic device, comprising: the system comprises a first memory and a first processor, wherein the first processor and the first memory are communicated with each other through a first bus; the first memory stores program instructions executable by the first processor, the first processor invoking the program instructions to perform a method comprising: determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
In a sixth aspect, an embodiment of the present invention further provides an electronic device, including: the second processor and the second memory complete communication with each other through a second bus; the second memory stores program instructions executable by the second processor, the second processor invoking the program instructions to perform the method of: loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory; and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
In a seventh aspect, the present invention provides a first computer readable storage medium having stored thereon a first computer program which, when executed by a processor, implements a method comprising: determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
In an eighth aspect, an embodiment of the present invention further provides a second computer-readable storage medium, on which a second computer program is stored, where the second computer program, when executed by a processor, implements the following method: loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory; and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
The method and the device for synchronizing the disk state information provided by the invention store the centralized state information such as the data block synchronization state, the data block synchronization state and the data block synchronization state, and the dispersed state information such as the data block state and the like by dividing the disk state information into the data block state, the data block group synchronization state and the data block synchronization state, so that when the disk state information is synchronized between a mechanical disk and a computer memory, the seek overhead during the synchronization of the disk state information is reduced, and the data writing efficiency is improved.
Drawings
FIG. 1 is a diagram illustrating a data structure of disk state information according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for synchronizing disk state information from a computer memory to a mechanical disk according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a method for loading disk state information from a mechanical disk into a computer memory according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an apparatus for synchronizing disk state information from a computer memory to a mechanical disk according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of an apparatus for loading disk state information from a mechanical disk into a computer memory according to an embodiment of the present invention;
fig. 6 is a block diagram of an electronic device according to an embodiment of the present invention;
fig. 7 is a block diagram of an electronic device according to another embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic diagram of a data structure of disk state information according to an embodiment of the present invention. As shown in FIG. 1, the space of the mechanical disk includes a state bitmap space and a data block space; the data block space is composed of a plurality of "data block groups" (fig. 1 shows only 1 data block group as an example), each data block group is composed of a preset number of consecutive data blocks, and each data block stores "metadata" and "user data". The metadata of the data block stores the state information of the user data; the first disk 'state bitmap space' is composed of a 'data block state bitmap space' and a 'data block group synchronization state bitmap space'; the data block state bitmap space stores the metadata of all data blocks of the data block space in a bitmap form, and the data block group synchronization state bitmap space stores the synchronization state between the metadata of all data block groups and the data block state bitmap in a bitmap form; the state of the data block is combined with a specific application scene, different application scenes have different state tables, and the state tables are used for indicating the mapping relation between the value of the metadata and the state of the data block.
For example, the device can be abstractly classified into an uninitialized state, an initialized state, an intermediate state and a final state. In various abstract states, the initialization state of a certain data block is uniquely determined, and the block in the initialization state can be reinitialized as an uninitialized block; the intermediate state may be divided into various sub-states; the final state does not transition to the other states.
The magnetic disk in the invention refers to a mechanical magnetic disk and does not include an SSD disk. The data block refers to a section of continuous space on the disk, and the size of the same data block is generally fixed. For example, in practical applications of the present invention, the size of the data block is set to a fixed 1 MB. The combination of a data block and the distributed status information corresponding to the data block is called chunk. The chunks in the disk are sequentially arranged according to chunk-ids, and the chunk-ids represent serial number identifiers of the data blocks in the disk.
FIG. 2 is a flowchart illustrating a method for synchronizing disk state information from a computer memory to a mechanical disk according to the present invention. As shown in fig. 2, on the basis of the above-mentioned disk state information data structure, the method includes the following steps:
step 201, determining metadata of each data block in a memory at the current moment based on an operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating a state of the user data in the data block;
specifically, the value of the metadata in each data block at the current time is determined in real time based on operations such as reading and writing of the data part in the data block of the disk, and the change of the state information of the data block is determined according to the value of the metadata.
Step 202, if it is determined that the user data of the data block at the current time is in an intermediate state or a final state, and a value of a data block group synchronization state of a data block group corresponding to the data block at the current time in the memory is a first preset value, and a value of a data block group synchronization state of the data block group at the next time in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space;
specifically, if the state of the data block at the current time is an "intermediate state" or a "final state", and the synchronization state of the data block corresponding to the data block at the current time in the memory is true (that is, the value of the synchronization state is a first preset value), the synchronization state of the data block in the memory is modified to be false (that is, the value of the synchronization state is a second preset value), and the modified synchronization state of the data block is updated to the state bitmap space of the mechanical disk in real time.
For example, if the state of the data block at the previous time is the non-initialized state or the initialized state, and the user performs a write operation on the data block, the state of the data block at the current time may be in the intermediate state or the final state, that is, when the data block is changed from the non-initialized state or the initialized state to another state, the state of the data block group corresponding to the data block in the data block group synchronization state bitmap needs to be processed. However, once a certain data block is switched to another state due to a change of user data, the synchronization state of the data block group corresponding to the data block must be set to false, because the data block is already switched from the "missing and unknown" state to another "uncertain" state.
Step 203, synchronizing the data block status bitmap of the data block in the memory to the status bitmap space every preset time.
Specifically, the preset time period may be set to be longer, for example, 1 minute, so as to reduce the average overhead of the synchronization frequency and the synchronization seek.
For example, for a certain disk initialization procedure, it is assumed that the state information of a data block can be roughly classified into the following three types: uninitialized State (uninized State), Initialized State (Initialized State), and written State (Final State). At this time, the metadata corresponding to each data block needs to use two bits to represent the state of the data block.
When querying a data block, if the contents of the data block are all uncertain and meaningless at this time, the metadata may use a first value "00" to indicate an uninitialized state.
When querying the data block, if the contents of the data block are all meaningful at this time, and the contents are set to the initial value, such as all 0 s or all 1 s, the metadata may use a second value "01" to indicate that the data block is in the initial state.
When querying a data block, if the content of the data block does not change, the metadata may use a third value "11" to indicate that the data block is in a written state.
According to the storage method, when the state of the data block changes, on one hand, the storage position of the metadata of the data block is adjacent to the storage of the user data, so that the metadata of the data block can be updated in real time, and the seek overhead is reduced; on the other hand, the data block state bitmap and the data block group synchronization state bitmap collectively store the state information of the data blocks, so that the memory can be quickly loaded; in addition, the amount of first disk state information can be controlled by defining the size of the data block, thereby controlling space overhead.
Optionally, the number of bits occupied by the data block group synchronization state bitmap and the number of data blocks corresponding to each bit are determined based on a preset compression ratio between the data block group synchronization state bitmap information and the data block state bitmap and the number of data blocks included in the data block space.
Specifically, a preset compression ratio is set between the data block group synchronization status bitmap and the data block status bitmap, that is, the data block group synchronization status bitmap roughly stores the synchronization status between the data block status bitmap and the metadata of the data block group, that is, one bit of the data block group synchronization status bitmap is used to represent the synchronization status of a group of "data block status bitmaps", so that the synchronization information is used to accelerate the synchronization speed between the "data block status bitmap" and the "metadata of the data block group" in some cases, for example, when the status of the data block is switched from "uninitialized status" to "initialized status", since the data block status bitmap is updated at regular time based on the data storage status in the data block, there is a data block status bitmap update delay condition, the "metadata of the data block" stored dispersedly has to be queried, but this results in slow loading; at this time, if the data block group synchronization state bitmap is in a synchronized state, the data block state bitmap can be considered as the real state of the data block, and the "metadata of the data block" stored in a scattered manner does not need to be queried.
It can be seen from the above embodiments that, the synchronization method can accelerate the synchronization speed of the data block status information and accelerate the data writing speed to a certain extent by setting the compression ratio between the data block group synchronization status bitmap and the data block status bitmap.
Fig. 3 is a schematic flowchart of a method for loading disk state information from a mechanical disk into a computer memory according to an embodiment of the present invention, where as shown in fig. 3, the method includes the following steps:
s301, loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory;
specifically, the disk state information includes a data block state, a data block group synchronization state, and a data block synchronization state, and the data block state and the data block group synchronization state are stored in the mechanical disk in a bitmap form.
S302, according to the data block state bitmap and the data block group synchronization state bitmap, a data block synchronization state bitmap is created in the computer memory.
Specifically, the state in the corresponding "data block synchronization state bitmap" is determined by traversing the data block state bitmap in the computer memory according to the state information of each data block and the state of the data block group corresponding to each data block in the data block synchronization state bitmap.
On the basis of the above embodiment, the method further includes:
when accessing a data block, inquiring the state of the data block in the data block synchronization state bitmap;
if the state of the data block in the data block synchronization state bitmap is not synchronous, loading metadata of a corresponding data block in a data block space in a mechanical disk into a memory;
and determining the state of the corresponding data block in the data block state bitmap in the computer memory according to the metadata, and updating the corresponding bit of the data block synchronization state bitmap in the computer memory to be synchronized.
On the basis of the foregoing embodiments, the creating a data block synchronization status bitmap in the computer memory according to the data block status bitmap and the data block group synchronization status bitmap in the method specifically includes:
if the state of the data block in the data block state bitmap is a final state, updating the state of the data block in the data block synchronization state bitmap to be synchronized;
and if the state of the data block in the data block state bitmap is an uninitialized state or an initialized state, and the state of the data block group corresponding to the data block in the data block group synchronization state bitmap is synchronized, updating the state of the data block in the data block synchronization state bitmap to be synchronized.
Specifically, 1) if the state of the data block in the "data block state bitmap" is "final state", the state of the data block in the "data block synchronization state bitmap" is set to "synchronized" (true). 2) If the state of the data block in the data block state bitmap is in an ' uninitialized ' or ' initialized ' state, and the value of the corresponding bit of the ' data block group ' corresponding to the ' data block ' in the ' data block group synchronization state bitmap is ' synchronized ' (true), the value of the corresponding bit of the data block in the ' data block synchronization state bitmap ' is set to ' synchronized ' (true). c) Otherwise, the state of the data block in the "data block synchronization state bitmap" is set to "out of synchronization" (false).
For convenience of description, the data block status bitmap in the present invention is also referred to as a first level bitmap, the data block group synchronization status bitmap is also referred to as a second level bitmap, and the metadata of the data block is also distributed status information.
For example, the following is a specific embodiment of bitmap initialization according to the method provided by the present invention, and the steps are as follows:
firstly, generating summary information, which mainly comprises: the method comprises the following steps of obtaining a magic value, a check value, the disk capacity, the disk serial number, the size of a data block, the number of data blocks in the disk, the offset of summary information on the disk, the offset of a first-level bitmap and a second-level bitmap on the disk and the like, wherein the disk capacity and the disk serial number are inherent attribute information of the disk, and the rest information can be set according to the disk capacity and the disk serial number.
Secondly, generating distributed state information, and setting the distributed state information to be in an uninitialized state;
thirdly, generating information of the first-stage bitmap, and setting the states of the first-stage bitmap to be uninitialized states;
and fourthly, generating the information of the second-level bitmap, wherein all the distributed state information is not initialized at the moment, and the state in the first-level bitmap is also not initialized, so that the states in the second-level bitmap are set to be the synchronized state.
And fifthly, writing the summary information, the distributed state information, the first-level bitmap and the second-level bitmap into a disk.
For example, the following is an embodiment of updating corresponding disk state information when the state of a data block on a disk changes, and the specific steps are as follows:
assuming that all 1000 data blocks in the data block space are written with "1", the content of the distributed state information of the 1000 data blocks is changed from the uninitialized state to the initialized state; according to the chunk-id of the chunk, the timer drives the first-level bitmap of the data block to update at regular time, and correspondingly updates from the uninitialized state to the initialized state in sequence; the state of the corresponding bit in the second-level bitmap is set to "1" at this time, which indicates that the first-level bitmap corresponding to the bit is the same as the corresponding distributed state information, and the state in the first-level bitmap can be considered as the state of the real data block.
Before the timer updates the first-stage bitmap next time, the data blocks in the data block space are written with data again, and then the distributed state information is from the initialized state to the written state.
Since the first-level bitmap is still in the initialized state and has not been updated yet, and the distributed state information has been immediately updated to the written state, the corresponding bits in the second-level bitmap are set out of synchronization.
And finally, storing the distributed state information, the first-stage bitmap and the second-stage bitmap.
It should be noted that, if the distributed status information is changed from one writing status to another writing status, the storage method of the disk status information is consistent with the above-mentioned embodiment.
For example, if the data on the disk managed by using the method provided by the present invention needs to be destroyed, only 0 needs to be written in the summary information part on the disk when the data is destroyed. When the summary information of the disk is read again, the disk is considered not to store the bitmap because the magic value and the check value are not consistent, and therefore the creation process is used for covering the existing data.
It can be seen that the synchronization method provided by the present invention can realize an initialization procedure at a disk level. For RAID5, after creation, the data in the disks needs to be written to 0. If the state is saved at the RAID level, if RAID initialization has been completed, and the RAID group is removed and several disks are added to create a new RAID5, then all disks need to be rewritten with 0. Such repeated operations can be avoided if a disk-level initialization procedure is used. And the initialization at the disk level can identify that the data blocks are initialized, so that the IO of the disk is further reduced, and thus the initialization speed of the whole RAID group and the service life of the disk are improved.
For example, based on the synchronization method provided by the present invention, a synchronization bitmap may be preset in the memory, and the synchronization bitmap is used to record the synchronization status of the first-level bitmap and the distributed status information in detail. When a disk is added, bitmap information stored in the disk needs to be loaded into a memory, and the method can be carried out according to the following steps:
firstly, reading bitmap summarizing information in a disk and checking the validity of the bitmap summarizing information. If the bitmap data is legal, the following bitmap loading process is continued, and if the bitmap data is illegal, bitmap initialization can be carried out, and new bitmap data is created.
Secondly, in the memory, the synchronous bitmaps are all set to be not synchronous.
And thirdly, loading the first-stage bitmap and the second-stage bitmap into the memory.
Fourthly, traversing the first-level bitmap, and executing the following process for each chunk:
if the first bitmap State of chunk is Final State, the value of the bit in the corresponding synchronous bitmap is set to 1.
If the first bitmap State of the chunk is not the Final State, but the bit in the second level bitmap corresponding to the chunk is 1 (indicating synchronization), the value of the bit in the corresponding synchronization bitmap is set to 1.
And fifthly, checking whether a bit of 0 exists in the synchronous bitmap, if so, starting a timer, sequentially loading the distributed state information to the memory, and loading the corresponding position 1 in the synchronous bitmap.
It can be seen from the above loading process that, based on the storage method provided by the present invention, since the centralized first-level bitmap and the centralized second-level bitmap are loaded first, the state of most data blocks is loaded, and then the rest of the distributed state information is loaded according to the situation, the speed of loading the bitmaps into the memory can be increased.
In the process of using bitmap, the state of the data block is checked, and the data block is accessed according to a certain mode. After the bitmap is loaded, not all chunks are necessarily in a synchronous State in the synchronous bitmap, nor are all data blocks necessarily in an Initialized State.
Based on the storage method provided by the invention, when accessing the data block, the following steps are required:
firstly, checking the state of chunk in the synchronous bitmap, if the chunk is not synchronized, considering that the state stored in the first-level bitmap is invalid, at this time, reading the distributed state information, loading the distributed state information into the memory, and then executing subsequent operations.
Second, check the state of chunk in the first level bitmap
If it is Uninitialized State, it is sufficient to directly return the result of initializing the data block (e.g. all 0 s) during reading, and it is not necessary to read the disk. When writing data, it is necessary to perform initialization of the corresponding data and the distributed status information first, and then perform subsequent operations.
If it is Initialized State, it is sufficient to directly return the result of initializing the data block (e.g. all 0 s) during reading, and it is not necessary to read the disk. When writing data, it is necessary to update the distributed State information to the Other State first, and then write the data.
If it is the Other State. At the time of reading, data in the disk is read. At the time of writing, an appropriate operation is performed according to the State of Other State. In the simplest case, there is only one Other State: if the data is written, the data is directly written into the disk, and the Other State does not need to be updated.
It can be seen that the entire access process to the data block is asynchronous because of the large number of disk reads and writes involved. Therefore, there are many access requests for the same data block, and in order to ensure the consistency of the states, a table of the currently accessed data block can be maintained in the memory, the access state is saved, and the access conflict is checked.
Fig. 4 is a schematic structural diagram of an apparatus for synchronizing disk state information from a computer memory to a mechanical disk according to an embodiment of the present invention. As shown in fig. 4, the apparatus includes: determination section 401, determination section 402, and synchronization section 403. Wherein:
the determining unit 401 determines, in real time, metadata of each data block in the memory at a current time based on an operation type of a user on user data in each data block by the user, where the metadata is used to indicate a state of the user data in the data block; if the determining unit 402 determines that the user data of the data block at the current time is in the intermediate state or the final state, and a value of a data block group synchronization state of a data block group corresponding to the data block in the memory at the current time is a first preset value, and a value of a data block group synchronization state of the data block group in the memory at the next time is a second preset value, the data block group synchronization state is updated to the state bitmap space; the synchronization unit 403 synchronizes the data block status bitmap of the data block in the memory to the status bitmap space every preset time.
Specifically, the determining unit 401 determines a value of metadata in each data block at the current time based on operations such as reading and writing of a data portion in a data block of a disk, determines a change of state information of the data block according to the value of the metadata, and sends a determination result to the determining unit 402; if the determining unit 402 determines that the state of the data block at the current time is "intermediate state" or "final state", and the synchronization state of the data block corresponding to the data block in the memory at the current time is true (that is, the value of the synchronization state is a first preset value), the synchronization state of the data block in the memory is modified to be false (that is, the value of the synchronization state is a second preset value), and the modified synchronization state of the data block is updated to the state bitmap space of the mechanical disk in real time; the synchronization unit 403 is responsible for synchronizing the data block status bitmap of the data block in the memory to the status bitmap space every preset time. It should be noted that the apparatus provided in the embodiment of the present invention is for implementing the method embodiment, and the functions of the apparatus may specifically refer to the method embodiment, which is not described herein again.
Fig. 5 is a schematic structural diagram of an apparatus for loading disk state information from a mechanical disk into a computer memory according to an embodiment of the present invention. As shown in fig. 5, the apparatus includes: a loading unit 501 and a creating unit 502. Wherein:
the loading unit 501 loads the data block status bitmap and the data block group synchronization status bitmap in the mechanical disk into the computer memory; the creating unit 502 creates a data block synchronization status bitmap in the computer memory according to the data block status bitmap and the data block group synchronization status bitmap.
Specifically, the disk state information includes a data block state, a data block group synchronization state, and a data block synchronization state, and the data block state and the data block group synchronization state are stored in the mechanical disk in a bitmap form. The loading unit 501 loads the disk state information stored in the mechanical disk into the memory of the computer, and sends the loading result to the creating unit 502; the creating unit 502 traverses the data block status bitmap in the computer memory according to the loading result, and determines the status in the corresponding "data block synchronization status bitmap" according to the status information of each data block and the status of the data block group corresponding to each data block in the data block synchronization status bitmap.
Fig. 6 is a block diagram of an electronic device according to an embodiment of the present invention. Referring to fig. 6, the electronic device includes: a first processor (processor)601, a first memory (memory)602, and a first bus 603; wherein the first processor 601 and the first memory 602 communicate with each other via the first bus 603; the first processor 601 is configured to call program instructions in the first memory 602 to perform the methods provided by the above-mentioned method embodiments, for example, including: determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
Fig. 7 is a block diagram of an electronic device according to another embodiment of the present invention. Referring to fig. 7, the electronic device includes: a second processor (processor)701, a second memory (memory)702, and a second bus 703; the second processor 701 and the second memory 702 complete communication with each other through the second bus 703; the second processor 701 is configured to call the program instructions in the second memory 702 to execute the method provided by the above-mentioned method embodiments, for example, including: loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory; and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
Embodiments of the present invention provide a computer program product comprising a first computer program stored on a non-transitory first computer readable storage medium, the first computer program comprising program instructions, which when executed by a computer, enable the computer to perform the methods provided by the above-mentioned method embodiments, for example, including: determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
Embodiments of the present invention also provide a computer program product comprising a second computer program stored on a non-transitory second computer-readable storage medium, the second computer program comprising program instructions, which when executed by a computer, enable the computer to perform the methods provided by the above-mentioned method embodiments, for example, including: loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory; and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
An embodiment of the present invention provides a first computer-readable storage medium, where the first computer-readable storage medium stores computer instructions, where the computer instructions cause the computer to execute the methods provided by the foregoing method embodiments, for example, including: determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block; if the user data of the data block at the current moment is in the intermediate state or the final state, and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, and the value of the data block group synchronization state of the data block group at the next moment in the memory is a second preset value, the data block group synchronization state is updated to the state bitmap space; and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
An embodiment of the present invention further provides a second computer-readable storage medium, where the second computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the foregoing method embodiments, for example, including: loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory; and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
It should be noted that the storage device provided by the present invention is for implementing the above method embodiment, and the functions thereof may specifically refer to the above method embodiment, which is not described herein again.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for synchronizing disk state information from a computer memory to a mechanical disk is characterized in that the space of the mechanical disk comprises a data block space and a state bitmap space; the data block space is composed of a plurality of data block groups, each data block group is composed of a preset number of continuous data blocks, each data block stores metadata and user data, the disk state information includes a data block state, a data block group synchronization state and a data block synchronization state, and the data block state, the data block group synchronization state and the data block synchronization state are stored in the computer memory in a bitmap form, the method includes:
determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block;
if the user data of the data block at the current time is in the intermediate state or the final state and the value of the data block group synchronization state of the data block group corresponding to the data block at the current time in the memory is a first preset value, modifying the data block group synchronization state of the data block group in the memory to a second preset value, and updating the modified data block group synchronization state to the state bitmap space;
and synchronizing the data block state bitmap of the data block in the memory to the state bitmap space every other preset time length.
2. A method of loading disk state information from a mechanical disk into a computer memory, the disk state information comprising a data block state, a data block group synchronization state, and a data block synchronization state, the data block state and the data block group synchronization state being stored in the mechanical disk in a bitmap form, the method comprising:
loading the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory;
and creating a data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
3. The method of claim 2, further comprising:
when accessing a data block, inquiring the state of the data block in the data block synchronization state bitmap;
if the state of the data block in the data block synchronization state bitmap is not synchronous, loading metadata of a corresponding data block in a data block space in a mechanical disk into a memory;
and determining the state of the corresponding data block in the data block state bitmap in the computer memory according to the metadata, and updating the corresponding bit of the data block synchronization state bitmap in the computer memory to be synchronized.
4. The method according to claim 2 or 3, wherein a data block synchronization status bitmap is created in the computer memory according to the data block status bitmap and the data block group synchronization status bitmap, specifically:
if the state of the data block in the data block state bitmap is a final state, updating the state of the data block in the data block synchronization state bitmap to be synchronized;
and if the state of the data block in the data block state bitmap is an uninitialized state or an initialized state, and the state of the data block group corresponding to the data block in the data block group synchronization state bitmap is synchronized, updating the state of the data block in the data block synchronization state bitmap to be synchronized.
5. An apparatus for synchronizing disk state information from a computer memory to a mechanical disk, wherein the space of the mechanical disk comprises a data block space and a state bitmap space; the data block space is composed of a plurality of data block groups, each data block group is composed of a preset number of continuous data blocks, each data block stores metadata and user data, the disk state information includes a data block state, a data block group synchronization state and a data block synchronization state, and the data block state, the data block group synchronization state and the data block synchronization state are stored in the computer memory in a bitmap form, the apparatus includes:
the determining unit is used for determining metadata of each data block in a memory at the current moment based on the operation type of a user on user data in each data block in real time, wherein the metadata is used for indicating the state of the user data in the data block;
if the judging unit judges that the user data of the data block at the current moment is in an intermediate state or a final state and the value of the data block group synchronization state of the data block group corresponding to the data block at the current moment in the memory is a first preset value, the judging unit modifies the data block group synchronization state of the data block group in the memory into a second preset value and updates the modified data block group synchronization state to the state bitmap space;
and the synchronization unit synchronizes the data block state bitmap of the data block in the memory to the state bitmap space every preset time length.
6. An apparatus for loading disk state information from a mechanical disk into a computer memory, the disk state information comprising a data block state, a data block group synchronization state, and a data block synchronization state, the data block group synchronization state, and the data block synchronization state being stored in the mechanical disk in a bitmap form, the apparatus comprising:
the loading unit loads the data block state bitmap and the data block group synchronization state bitmap in the mechanical disk into the computer memory;
and the creating unit is used for creating the data block synchronization state bitmap in the computer memory according to the data block state bitmap and the data block group synchronization state bitmap.
7. The apparatus of claim 6, further comprising:
the access unit is specifically used for inquiring the state of the data block in the data block synchronization state bitmap when the data block is accessed; if the state of the data block in the data block synchronization state bitmap is not synchronous, loading metadata of a corresponding data block in a data block space in a mechanical disk into a memory; and determining the state of the corresponding data block in the data block state bitmap in the computer memory according to the metadata, and updating the corresponding bit of the data block synchronization state bitmap in the computer memory to be synchronized.
8. The apparatus according to claim 6, wherein the creating unit specifically includes:
the first synchronization subunit updates the state of the data block in the data block synchronization state bitmap to be synchronized if the state of the data block in the data block state bitmap is a final state;
and the second synchronization subunit updates the state of the data block in the data block synchronization state bitmap to be synchronized if the state of the data block in the data block state bitmap is an uninitialized state or an initialized state and the state of the data block group corresponding to the data block in the data block synchronization state bitmap is synchronized.
9. An electronic device, comprising a memory and a processor, wherein the processor and the memory communicate with each other via a bus; the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1 to 4.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711086203.7A CN107797770B (en) | 2017-11-07 | 2017-11-07 | Method and device for synchronizing disk state information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711086203.7A CN107797770B (en) | 2017-11-07 | 2017-11-07 | Method and device for synchronizing disk state information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107797770A CN107797770A (en) | 2018-03-13 |
| CN107797770B true CN107797770B (en) | 2020-08-21 |
Family
ID=61547873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711086203.7A Active CN107797770B (en) | 2017-11-07 | 2017-11-07 | Method and device for synchronizing disk state information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107797770B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108536393B (en) * | 2018-03-20 | 2021-03-19 | 深圳神州数码云科数据技术有限公司 | Disk initialization method and device |
| CN111159142B (en) * | 2018-11-07 | 2023-07-14 | 马上消费金融股份有限公司 | Data processing method and device |
| CN110688071A (en) * | 2019-09-29 | 2020-01-14 | 上海英方软件股份有限公司 | Data synchronization method and system for reducing data synchronization quantity |
| CN112286454B (en) * | 2020-10-27 | 2022-07-22 | 苏州浪潮智能科技有限公司 | Bitmap synchronization method and device, electronic equipment and storage medium |
| CN113360095B (en) * | 2021-06-04 | 2023-02-17 | 重庆紫光华山智安科技有限公司 | Hard disk data management method, device, equipment and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1955939A (en) * | 2006-10-13 | 2007-05-02 | 清华大学 | Backup and recovery method based on virtual flash disk |
| CN101566924A (en) * | 2008-04-23 | 2009-10-28 | 中兴通讯股份有限公司 | Data read-write method of disk array |
| CN102096616A (en) * | 2011-02-15 | 2011-06-15 | 成都市华为赛门铁克科技有限公司 | Logical object data duplication method and related device |
| CN102567214A (en) * | 2011-12-01 | 2012-07-11 | 浪潮电子信息产业股份有限公司 | Method for quickly initializing bitmap page management by RAID5 |
| CN104881371A (en) * | 2015-05-29 | 2015-09-02 | 清华大学 | Persistent internal memory transaction processing cache management method and device |
| CN105808154A (en) * | 2014-12-31 | 2016-07-27 | 北京神州云科数据技术有限公司 | Bit map based dual-controller cache memory write-back method and apparatus |
| WO2017041616A1 (en) * | 2015-09-08 | 2017-03-16 | 中兴通讯股份有限公司 | Data reading and writing method and device, double active storage system and realization method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007241539A (en) * | 2006-03-07 | 2007-09-20 | Hitachi Systems & Services Ltd | Data management and control system for semiconductor flash memory, and semiconductor flash memory housing device |
-
2017
- 2017-11-07 CN CN201711086203.7A patent/CN107797770B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1955939A (en) * | 2006-10-13 | 2007-05-02 | 清华大学 | Backup and recovery method based on virtual flash disk |
| CN101566924A (en) * | 2008-04-23 | 2009-10-28 | 中兴通讯股份有限公司 | Data read-write method of disk array |
| CN102096616A (en) * | 2011-02-15 | 2011-06-15 | 成都市华为赛门铁克科技有限公司 | Logical object data duplication method and related device |
| CN102567214A (en) * | 2011-12-01 | 2012-07-11 | 浪潮电子信息产业股份有限公司 | Method for quickly initializing bitmap page management by RAID5 |
| CN105808154A (en) * | 2014-12-31 | 2016-07-27 | 北京神州云科数据技术有限公司 | Bit map based dual-controller cache memory write-back method and apparatus |
| CN104881371A (en) * | 2015-05-29 | 2015-09-02 | 清华大学 | Persistent internal memory transaction processing cache management method and device |
| WO2017041616A1 (en) * | 2015-09-08 | 2017-03-16 | 中兴通讯股份有限公司 | Data reading and writing method and device, double active storage system and realization method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107797770A (en) | 2018-03-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107797770B (en) | Method and device for synchronizing disk state information | |
| US20220188003A1 (en) | Distributed Storage Method and Device | |
| CN108427538B (en) | Storage data compression method and device of full flash memory array and readable storage medium | |
| US11531482B2 (en) | Data deduplication method and apparatus | |
| US20190163591A1 (en) | Remote Data Replication Method and System | |
| CN107229420B (en) | Data storage method, reading method, deleting method and data operating system | |
| US11580162B2 (en) | Key value append | |
| JP2019135637A (en) | Information processing method, information processing device, server, and computer-readable recording medium | |
| JP2014528127A (en) | Efficient access to storage devices with usage bitmaps | |
| US10628298B1 (en) | Resumable garbage collection | |
| CN106326229B (en) | File storage method and device of embedded system | |
| US20210081388A1 (en) | Methods, apparatuses and computer program products for managing metadata of storage object | |
| US11449402B2 (en) | Handling of offline storage disk | |
| CN112182010B (en) | Dirty page refreshing method and device, storage medium and electronic equipment | |
| CN107423425B (en) | Method for quickly storing and inquiring data in K/V format | |
| CN107992763B (en) | Power failure protection method and device for file system | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN110019527B (en) | Slave library reading method, related device and equipment | |
| KR20150035876A (en) | Method for de-duplicating data and apparatus therefor | |
| CN112579550B (en) | Metadata information synchronization method and system of distributed file system | |
| CN117348968A (en) | Cache data acceleration method, device and equipment of virtual disk | |
| CN111061698A (en) | Storage method and device of Ether house contract data | |
| CN114328007B (en) | Container backup and restoration method, device and medium thereof | |
| CN110716696A (en) | Object processing method and related device | |
| US11341007B2 (en) | Storage system and storage control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220324 Address after: No. 407, floor 4, No. 9, No. 9, shangdijiu street, Haidian District, Beijing 100085 Patentee after: Shenzhou Yunke (Beijing) Technology Co.,Ltd. Address before: 518131 F3, 11th floor, No. 8 Kefa Road, Yuehai Street, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen science and Technology Co.,Ltd. digital cloud data |
|
| TR01 | Transfer of patent right |