CN117349381A - Data synchronization method, device, management equipment, controller, system and medium - Google Patents
Data synchronization method, device, management equipment, controller, system and medium Download PDFInfo
- Publication number
- CN117349381A CN117349381A CN202311329650.6A CN202311329650A CN117349381A CN 117349381 A CN117349381 A CN 117349381A CN 202311329650 A CN202311329650 A CN 202311329650A CN 117349381 A CN117349381 A CN 117349381A
- Authority
- CN
- China
- Prior art keywords
- controller
- bitmap
- data
- main controller
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000013507 mapping Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 14
- 238000005192 partition Methods 0.000 claims description 13
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 13
- 238000004891 communication Methods 0.000 description 10
- 230000004044 response Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 101100476924 Caenorhabditis elegans sdc-1 gene Proteins 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001052 transient effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data synchronization method, a device, a management device, a controller, a system and a medium, wherein the method is applied to the management device and can comprise the following steps: setting a database and bitmaps corresponding to the controllers one by one in the storage equipment; setting a main controller to enable the main controller to independently obtain the read-write permission of the database, and updating the data writing position to the bitmap of the main controller when executing the data writing task to the database; setting a takeover controller when determining that the master controller fails, so that the takeover controller takes over the read-write permission of the database, and synchronizing data in a bitmap of the master controller to the bitmap of the takeover controller, so as to resume execution of the data writing service which is not completed by the master controller according to the bitmap of the takeover controller; the method can set the database shared by all controllers, and can ensure that the taking-over controller can restore the service only by synchronizing the bitmap information which is generated by the main controller and records the writing condition of the database, thereby improving the stability of the multi-control storage array.
Description
Technical Field
The present invention relates to the field of storage, and in particular, to a data synchronization method, apparatus, management device, controller, system, and medium.
Background
With the explosive growth of data and the development of storage technology, a multi-control storage array gradually becomes a storage mode for improving the system performance and the data reliability. The multi-control storage array typically includes a plurality of controllers that can store storage resource information of users, hard disks, storage pools, logical volumes, etc. in the storage system in a database to implement management of the multi-control storage array.
In the related art, the database files of each controller are usually stored in the system disk corresponding to each controller, and because the system disks among the multiple controllers are isolated from each other, a data synchronization means is needed to synchronize the database files of each controller. This results in that as the number of controllers increases, the data synchronization period and data synchronization algorithm become more complex, greatly increasing the risk of data loss; more seriously, if database synchronization is performed when the main controller fails, the database acquired by the main controller and the standby controller are inconsistent.
Disclosure of Invention
The invention aims to provide a data synchronization method, a device, a management device, a controller, a system and a medium, wherein a database shared by all controllers can be set, and the taking over controller can be ensured to recover service only by synchronizing bitmap information which is generated by a main controller and records the writing condition of the database, so that the stability of a multi-control storage array can be improved.
In order to solve the above technical problems, the present invention provides a data synchronization method, applied to a management device, including:
setting a database and bitmaps corresponding to the controllers one by one in the storage equipment;
setting a main controller to enable the main controller to independently obtain the read-write permission of the database, and updating a data writing position into a bitmap of the main controller when executing a data writing task to the database;
and setting a takeover controller when the main controller is determined to be faulty, so that the takeover controller takes over the read-write permission of the database, and synchronizing data in the bitmap of the main controller to the bitmap of the takeover controller, so that the data writing service which is not completed by the main controller is restored to be executed according to the bitmap of the takeover controller.
Optionally, before setting the database and the bitmap corresponding to each controller one by one in the storage device, the method further includes:
selecting a plurality of hard disk devices in a storage system, and constructing a disk array by utilizing the plurality of hard disk devices, so that the disk array at least allows one hard disk device to be damaged without interrupting storage service;
and setting the disk array as the storage device.
Optionally, the constructing a disk array by using the plurality of hard disk devices includes:
partitioning a partition space with a preset size in each hard disk device;
and constructing the disk array by utilizing the partition space corresponding to each hard disk device.
Optionally, after setting up the takeover controller, further comprising:
and transmitting the unfinished data writing service of the main controller to the takeover controller, so that the takeover controller can determine the completion progress of the main controller on the data writing service according to the data writing service and the bitmap of the takeover controller after synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, and resume executing the unfinished data writing service of the main controller according to the completion progress.
Optionally, the method further comprises:
dividing a storage space in the storage device into a plurality of data blocks, and setting a one-to-one mapping relation between each bit in the bitmap and the data blocks;
and transmitting the mapping relation information recorded with the mapping relation to each controller so that the main controller updates the corresponding bit in the bitmap of the main controller according to the data block corresponding to the data writing position.
The invention also provides a data synchronization method, which is applied to the controller and comprises the following steps:
when the managed device is set as a main controller, the read-write permission of a database in the storage device is independently obtained, and when a data writing task is executed to the database, a data writing position is updated to a bitmap corresponding to the managed device in the storage device; the storage equipment is provided with bitmaps which are in one-to-one correspondence with the controllers;
when the management device is set as a takeover controller, taking over the read-write permission of the database, and synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the data, so that the data writing service which is not completed by the main controller is restored to be executed according to the bitmap corresponding to the data writing service.
Optionally, after being set as the takeover controller by the management device, the method further includes:
when the unfinished data writing service of the main controller issued by the management equipment is received and the synchronization of the data in the bitmap of the main controller to the bitmap corresponding to the main controller is completed, determining the completion progress of the main controller on the data writing service according to the data writing service and the bitmap corresponding to the main controller, and recovering and executing the unfinished data writing service of the main controller according to the completion progress.
Optionally, after synchronizing the data in the bitmap of the main controller to its corresponding bitmap, the method further includes:
and clearing the data in the bitmap of the main controller.
Optionally, the updating the data writing position to the bitmap corresponding to the data writing position in the storage device includes:
obtaining mapping relation information; the mapping relation information records a one-to-one mapping relation between each bit in the bitmap and each data block in the storage device;
and updating the corresponding bit in the bitmap according to the data block corresponding to the data writing position.
The invention also provides a data synchronization device, which is applied to the management equipment and comprises:
The storage device setting module is used for setting a database and bitmaps corresponding to the controllers one by one in the storage device;
the main controller setting module is used for setting a main controller so that the main controller can independently obtain the read-write permission of the database, and when a data writing task is executed to the database, the data writing position is updated to the bitmap of the main controller;
and the takeover controller setting module is used for setting a takeover controller when the main controller is determined to be faulty so that the takeover controller takes over the read-write permission of the database, and synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, so that the unfinished data writing service of the main controller is restored to be executed according to the bitmap of the takeover controller.
The invention also provides a data synchronization device, which is applied to a controller and comprises:
the main controller module is used for independently obtaining the read-write permission of a database in the storage device when the managed device is set as the main controller, and updating the data writing position to a bitmap corresponding to the main controller in the storage device when a data writing task is executed to the database; the storage equipment is provided with bitmaps which are in one-to-one correspondence with the controllers;
And the take-over module is used for taking over the read-write permission of the database and synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the main controller when the management equipment is set as the take-over controller so as to resume executing the unfinished data writing service of the main controller according to the bitmap corresponding to the main controller.
The present invention also provides a management apparatus comprising:
a memory for storing a computer program;
and a processor for implementing the data synchronization method applied to the management device as described above when executing the computer program.
The present invention also provides a controller comprising:
a memory for storing a computer program;
and a processor for implementing the data synchronization method applied to the controller as described above when executing the computer program.
The invention also provides a data synchronization system, comprising:
the storage device is used for setting a database and bitmaps corresponding to the controllers one by one;
a management device for executing the data synchronization method applied to the management device as described above;
and a controller for performing the data synchronization method applied to the controller as described above.
The present invention also provides a computer-readable storage medium having stored therein computer-executable instructions which, when loaded and executed by a processor, implement a data synchronization method applied to a management device as described above and/or a data synchronization method applied to a controller as described above.
The invention provides a data synchronization method, which is applied to management equipment and comprises the following steps: setting a database and bitmaps corresponding to the controllers one by one in the storage equipment; setting a main controller to enable the main controller to independently obtain the read-write permission of the database, and updating a data writing position into a bitmap of the main controller when executing a data writing task to the database; and setting a takeover controller when the main controller is determined to be faulty, so that the takeover controller takes over the read-write permission of the database, and synchronizing data in the bitmap of the main controller to the bitmap of the takeover controller, so that the data writing service which is not completed by the main controller is restored to be executed according to the bitmap of the takeover controller.
It can be seen that the management device in the present invention can firstly set a database in the storage device and a bitmap corresponding to each controller one by one, wherein the database is set only in one set and shared by each controller, and the bitmap is used for recording the update condition of the controllers to the database; subsequently, the management device may set the main controller so that the main controller individually obtains the read-write permission of the database, and updates the data writing position to the bitmap of the main controller when executing the data writing task to the database; when the fault of the master controller is determined, the management device can set the takeover controller so that the takeover controller takes over the read-write permission of the database and synchronizes the data in the bitmap of the master controller to the bitmap of the takeover controller, so that the unfinished data writing service of the master controller is restored and executed according to the bitmap of the takeover controller, namely, the takeover controller only needs to synchronize the bitmap data recorded by the master controller and can master the updating condition of the master controller on the database according to the bitmap data, thereby being capable of restoring and executing the unfinished data writing service of the master controller without completely synchronizing the database; and the bitmap data generated by the main controller is recorded on the storage device independent of the main controller, so that when the main controller fails, the takeover controller can still complete data synchronization, thereby greatly improving the stability of the multi-control storage array and reducing the risk of data loss. The invention also provides a data synchronization device, a management device, a controller, a system and a computer readable storage medium, which have the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of a data synchronization system according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data synchronization method according to an embodiment of the present invention;
FIG. 3 is a flowchart of another data synchronization method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of data distribution of a RAID1 device with redundancy protection function according to an embodiment of the present invention;
FIG. 5 is a block diagram of a data synchronization device according to an embodiment of the present invention;
FIG. 6 is a block diagram illustrating another data synchronization apparatus according to an embodiment of the present invention;
fig. 7 is a block diagram of a management device according to an embodiment of the present invention;
fig. 8 is a block diagram of a controller according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the related art, the database files of each controller are usually stored in the system disk corresponding to each controller, and because the system disks among the multiple controllers are isolated from each other, a data synchronization means is needed to synchronize the database files of each controller. This results in that as the number of controllers increases, the data synchronization period and data synchronization algorithm become more complex, greatly increasing the risk of data loss; more seriously, if database synchronization is performed when the main controller fails, the database acquired by the main controller and the standby controller are inconsistent. In view of this, the present invention can provide a data synchronization method, which can set a database shared by all controllers, and can ensure that the takeover controller can recover the service only by synchronizing the bitmap information generated by the main controller and recorded with the writing condition of the database, so as to improve the stability of the multi-control storage array.
For ease of understanding, please refer to fig. 1, fig. 1 is a block diagram of a data synchronization system according to an embodiment of the present invention. The system comprises a management device 10, a plurality of controllers 20 and a storage device 30, wherein the plurality of controllers 20 can form a cluster, and the management device 10 can centrally manage the cluster, for example, a main controller and a standby controller can be arranged in the controllers 20; the management device 10 is further configured to set a database in the storage device and a bitmap corresponding to each controller one by one, where the database is shared by all the controllers 20, and the bitmap is used to record the update of the database by the controllers 20. Specifically, the standby controller may perform a read operation on the database, while the main controller may perform a read operation and a write operation on the database. When writing data to the database, the host controller may further record the location of the written data in its corresponding bitmap. Furthermore, when the main controller fails, the writing condition of the main controller to the database can be determined according to the bitmap of the main controller, so that the takeover controller taking over the main controller can synchronize the bitmap data generated by the main controller to the bitmap of the main controller, and the unfinished writing service of the main controller can be recovered according to the bitmap data in the bitmap of the main controller. Obviously, in the embodiment of the present invention, since each controller 20 does not separately store the database file, but sets a database shared by all the controllers 20, and the database is set in a storage device independent of the controllers 20, it is avoided that the database file is synchronized between the controllers 20, and the situation that the takeover controller cannot acquire the database file of the master controller when the master controller fails is avoided; and when the service is recovered, the takeover controller only needs to synchronize the bitmap data generated by the main controller, and does not need to completely synchronize the database, so that the data synchronization quantity can be greatly reduced.
It should be noted that, in the embodiment of the present invention, the management device 10 and the controller 20 are not limited to be in the form of physical devices or virtual machines, and may be set according to actual application requirements. When the physical device exists, the embodiment of the present invention is not limited to the physical device specifically corresponding to the management device 10 and the controller 20, and may be, for example, a personal computer, a server, etc.; when present in the form of a virtual machine, embodiments of the present invention are not limited to the specific type of physical device in which the management device 10, the controller 20, or both the management device 10 and the controller 20 are deployed, and may be, for example, a personal computer, a server, or the like. The number of the controllers 20 is not limited, and the controller may be set according to actual application requirements.
Based on the description of the data synchronization system, the data synchronization method applied to the management device provided by the embodiment of the invention will be described in detail. Referring to fig. 2, fig. 2 is a flowchart of a data synchronization method according to an embodiment of the invention. The method is applied to the management device and can comprise the following steps:
and S201, setting a database in the storage device and a bitmap corresponding to each controller one by one.
In the embodiment of the present invention, the management device may first set a database and a bitmap (bitmap) corresponding to each controller in the storage device. Wherein the database is shared by all controllers, e.g., all controllers have read rights to the database, and the master controller further has read rights to the database; the bitmap is used for storing the update condition of the corresponding controller to the database. For the database, the controllers in the embodiment of the invention can not independently maintain the database files, but can use the same database to store important data (such as storage resource information of users, hard disks, storage pools, logical volumes and the like) in the multi-control storage array, so that the problems of poor stability and easy data loss caused by data synchronization among the controllers can be avoided. For the bitmap, each controller can independently maintain the corresponding bitmap, so that data confusion can be avoided; and because the host controller uses the corresponding bitmap to record the update condition of the database, the takeover controller of the takeover host controller can determine the write condition of the host controller to the database according to the bitmap of the host controller, so after the bitmap data generated by the host controller is synchronized to the bitmap of the host controller, the takeover controller can quickly recover the unfinished database write service of the host controller. More importantly, because the storage device is independent of each controller, when the controller is in fault, other controllers can still access the bitmap and the database from the independent storage device, so that the problems that data cannot be acquired and are inconsistent due to the controller fault can be avoided.
It should be noted that, the embodiment of the present invention is not limited to a specific manner of setting a database in a storage device, and reference may be made to a related art of a storage System, for example, the storage device may be formatted as an XFS File System (X File System), and a database File may be stored in the File System to form the database. To ensure that each controller has access to the database, the management device may also mount a data set path to each controller. The embodiment of the invention is not limited to a specific way of setting the bitmap in the storage device, and for example, a corresponding bitmap space may be allocated for each controller, so that each bit (bit) in the bitmap space is used as each bit in the bitmap.
Further, the storage device in the embodiment of the present invention may be composed of a single hard disk device or may be composed of a plurality of hard disk devices. In order to further improve the storage reliability of the database and the bitmap, the embodiment of the invention not only utilizes a plurality of hard disk devices to form the storage device, but also can utilize the hard disk device assembly as a disk array, so that the disk array at least allows one hard disk device to be damaged without interrupting the storage service, thereby ensuring that the database and the bitmap are not lost when part of the hard disk devices are damaged. It should be noted that, the embodiment of the present invention is not limited to a specific form of the array, for example, a plurality of hard disk device components RAID1, RAID5, and RAID10 arrays may be utilized, and all three arrays at least allow a piece of hard disk device to be damaged without interrupting the storage service, and may be selected according to practical application requirements. The embodiment of the invention is not limited to the specific number of the hard disk devices, and can be set according to the actual application requirements.
Based on this, before setting the database and the bitmap corresponding to each controller one by one in the storage device, it may further include:
step 11: selecting a plurality of hard disk devices in a storage system, and utilizing the plurality of hard disk devices to construct a disk array, so that the disk array at least allows one hard disk device to be damaged without interrupting storage service;
step 12: the disk array is set as a storage device.
Furthermore, because the space occupation of the database and the bitmap on the hard disk device is not large, in order to avoid resource waste, the embodiment of the invention can also only divide the partition space with the preset size in each hard disk device and construct the disk array by utilizing the partition space corresponding to each hard disk device. It should be noted that, the embodiment of the present invention is not limited to a specific preset size, and may be set according to practical application requirements, for example, may be 8G.
Based on this, constructing a disk array using a plurality of hard disk devices may include:
step 21: dividing partition spaces with preset sizes in each hard disk device;
step 22: and constructing a disk array by utilizing the partition space corresponding to each hard disk device.
S202, setting a main controller so that the main controller can independently obtain the read-write permission of the database, and updating the data writing position into the bitmap of the main controller when the data writing task is executed to the database.
After the database and bitmap setting is completed, the management device in the embodiment of the invention selects the main controller from all controllers so that the main controller can independently obtain the read-write authority of the database. And the master controller can update the data writing position to the bitmap of the master controller when performing the data writing task to the database.
In order to ensure that the controller can record the data writing position by using the bitmap, when the storage device is set, the management device can divide the storage space in the storage device into a plurality of data blocks and set a one-to-one mapping relation between each bit in the bitmap and the data blocks; then, the management device may issue the mapping relationship information recorded with the mapping relationship to each controller, so that the host controller updates the corresponding bit in the bitmap of the host controller according to the data block corresponding to the data writing position. It should be noted that when the storage device is presented in a disk array, the data block may be a chunk in a disk array, so as to correspond each bit in the bitmap to a basic storage unit for performing data recovery in the disk array, thereby facilitating data recovery.
Based on this, the present invention may further include:
Step 31: dividing a storage space in a storage device into a plurality of data blocks, and setting a one-to-one mapping relation between each bit in a bitmap and the data blocks;
step 32: and transmitting the mapping relation information recorded with the mapping relation to each controller so that the main controller updates the corresponding bit in the bitmap of the main controller according to the data block corresponding to the data writing position.
And S203, setting a takeover controller when the fault of the master controller is determined, so that the takeover controller takes over the read-write permission of the database, and synchronizing the data in the bitmap of the master controller to the bitmap of the takeover controller, so that the unfinished data writing service of the master controller is restored to be executed according to the bitmap of the takeover controller.
Upon determining that the primary controller fails, the management platform may select a take-over controller among the remaining normally operating standby controllers to take over the primary controller with the take-over controller. It should be noted in particular that the take-over controller is a special master controller. Before the takeover controller starts writing data into the database, the bitmap data generated by the main controller with faults need to be synchronized so as to master the writing update condition of the main controller to the database. After the data synchronization is completed, the takeover controller is formally changed into a new main controller, so that the incomplete data writing service of the main controller can be restored and executed according to the bitmap of the takeover controller, or when the fact that the prior main controller has no incomplete data writing service is determined, the response to the new data writing service is started.
Specifically, after the takeover controller is set, the management platform can issue the data writing service which is not finished by the main controller to the takeover controller, so that the takeover controller can determine the completion progress of the main controller on the data writing service according to the data writing service and the bitmap of the takeover controller after synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, and resume executing the data writing service which is not finished by the main controller according to the completion progress. Of course, the takeover controller can also obtain the unfinished data writing service of the main controller from the management platform by itself, and can be set according to the actual application requirements.
Furthermore, it should be noted that the embodiment of the present invention does not limit how the management platform detects whether the main controller fails, for example, a heartbeat mechanism may be set between the management platform and each controller, so that the working and failure conditions of each controller may be determined according to the condition that each controller responds to a heartbeat message.
Based on the above embodiment, the management device in the present invention may first set a database and a bitmap corresponding to each controller one by one in the storage device, where the database is set only in one set and shared by each controller, and the bitmap is used to record the update condition of the database by the controller; subsequently, the management device may set the main controller so that the main controller individually obtains the read-write permission of the database, and updates the data writing position to the bitmap of the main controller when executing the data writing task to the database; when the fault of the master controller is determined, the management device can set the takeover controller so that the takeover controller takes over the read-write permission of the database and synchronizes the data in the bitmap of the master controller to the bitmap of the takeover controller, so that the unfinished data writing service of the master controller is restored and executed according to the bitmap of the takeover controller, namely, the takeover controller only needs to synchronize the bitmap data recorded by the master controller and can master the updating condition of the master controller on the database according to the bitmap data, thereby being capable of restoring and executing the unfinished data writing service of the master controller without completely synchronizing the database; and the bitmap data generated by the main controller is recorded on the storage device independent of the main controller, so that when the main controller fails, the takeover controller can still complete data synchronization, thereby greatly improving the stability of the multi-control storage array and reducing the risk of data loss.
Based on the foregoing embodiments, a data synchronization method applied to a controller according to an embodiment of the present invention will be described in detail below. Referring to fig. 3, fig. 3 is a flowchart of another data synchronization method provided in an embodiment of the present invention, where the method is applied to a controller and may include:
s301, when the managed device is set as a main controller, reading and writing rights of a database in the storage device are obtained independently, and when a data writing task is executed to the database, a data writing position is updated to a bitmap corresponding to the managed device in the storage device; the storage device is provided with a bitmap corresponding to each controller one by one.
It is understood that the present controller is adjusted from the standby controller to the main controller. As described above, the storage device is provided with bitmaps corresponding to the controllers one by one, and these bitmaps are maintained independently by the controllers. Therefore, when the managed device is set as the master controller, the controller needs to update the data writing position to the bitmap corresponding to the controller in the storage device when executing the data writing task to the database.
In order to ensure that the controller can record the data writing position by using the bitmap, when the storage device is set, the management device can divide the storage space in the storage device into a plurality of data blocks and set a one-to-one mapping relation between each bit in the bitmap and the data blocks; then, the management device may issue the mapping relationship information recorded with the mapping relationship to each controller, so that the host controller updates the corresponding bit in the bitmap of the host controller according to the data block corresponding to the data writing position.
Based on this, updating the data writing location to the bitmap corresponding to itself in the storage device may include:
step 41: obtaining mapping relation information; the mapping relation information records a one-to-one mapping relation between each bit in the bitmap and each data block in the storage device;
step 42: and updating the corresponding bit in the bitmap according to the data block corresponding to the data writing position.
S302, when the managed device is set to take over the controller, taking over the read-write permission of the database, and synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the managed device so as to resume executing the unfinished data writing service of the main controller according to the bitmap corresponding to the managed device.
It is also understood that the present controller is adjusted from a standby controller to a take-over controller. As described above, the host controller needs to synchronize the bitmap data generated by the failed host controller before starting writing data to the database, so as to grasp the writing update condition of the host controller to the database. After the data synchronization is completed, the takeover controller is formally changed into a new main controller, so that the incomplete data writing service of the main controller can be restored and executed according to the bitmap of the takeover controller, or when the fact that the prior main controller has no incomplete data writing service is determined, the response to the new data writing service is started.
For the scene of recovering the data writing service, when the controller receives the unfinished data writing service issued by the management device and completes synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the controller, the controller can determine the completion progress of the main controller on the data writing service according to the data writing service and the bitmap corresponding to the main controller, and recover and execute the unfinished data writing service of the main controller according to the completion progress. Of course, the takeover controller can also obtain the unfinished data writing service of the main controller from the management platform by itself, and can be set according to the actual application requirements.
Further, after the synchronization of the bitmap data is completed, in order to avoid that the previously failed main controller is online again and reads the old bitmap data to be wrong, the takeover controller can also clear the data in the bitmap of the main controller after synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the takeover controller.
Based on the above embodiment, the management device in the present invention may first set a database and a bitmap corresponding to each controller one by one in the storage device, where the database is set only in one set and shared by each controller, and the bitmap is used to record the update condition of the database by the controller; subsequently, the management device may set the main controller so that the main controller individually obtains the read-write permission of the database, and updates the data writing position to the bitmap of the main controller when executing the data writing task to the database; when the fault of the master controller is determined, the management device can set the takeover controller so that the takeover controller takes over the read-write permission of the database and synchronizes the data in the bitmap of the master controller to the bitmap of the takeover controller, so that the unfinished data writing service of the master controller is restored and executed according to the bitmap of the takeover controller, namely, the takeover controller only needs to synchronize the bitmap data recorded by the master controller and can master the updating condition of the master controller on the database according to the bitmap data, thereby being capable of restoring and executing the unfinished data writing service of the master controller without completely synchronizing the database; and the bitmap data generated by the main controller is recorded on the storage device independent of the main controller, so that when the main controller fails, the takeover controller can still complete data synchronization, thereby greatly improving the stability of the multi-control storage array and reducing the risk of data loss.
The above data synchronization method will be described in detail based on specific examples.
1. And dividing 8G partitions by the first two hard disks in the multi-control storage system to form RAID1 equipment, and setting the first 512K space of the member disk of the RAID1 equipment as a bitmap space. Mainly comprises the following steps:
(1) The first two hard disks in the multi-control storage system are selected as member disks sdb and sdc of RAID1 equipment;
(2) Partitioning the sdb and the sdc to obtain partition spaces sdb1 and sdc1 with the size of 8G;
(3) Forming sdb1 and sdc1 into RAID1 equipment, dividing the RAID1 equipment into specified chunk sizes, and mapping 1bit of a bitmap with chunk, namely, 1bit in the bitmap corresponds to 1 chunk size in the RAID1 equipment;
(4) And recording the data change of RAID1 by using bitmap data, reserving and setting the front 512K space of the member disks sdb1 and sdc1 of the RAID1 equipment as a bitmap space, and having a data redundancy function.
2. And allocating a unique slot number for each controller in the multi-control storage system, and establishing a bitmap space with the size of 4K for each controller, wherein the bitmap numbers correspond to the slot numbers of the controllers one by one. Mainly comprises the following steps:
(1) Each controller in the multi-control storage system has a unique slot number, if the slot number of the controller 1 is 1, the slot number of the controller N is N;
(2) Each controller maintains a 4K bitmap data, namely, bitmap data corresponding to bitmapnode00N by a controller with slot number N, and records all writing operations and data changes on the controller.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating data distribution of a RAID1 device with redundancy protection according to an embodiment of the present disclosure. Therefore, in the sdb and the sdc of the two hard disks, the front 512K space is a bitmap space, each controller corresponds to a bitmap space with the size of 4K, and the bitmap numbers and the controller slot numbers are in one-to-one correspondence. At the same time, each bit in the bitmap corresponds to each data block in the partition.
3. Setting read-write priority of the controller for accessing RAID1 equipment, formatting the RAID1 equipment into an XFS file system, and starting database service. Mainly comprises the following steps:
(1) The main controller has the read-write authority of RAID1 equipment, and other alternative controllers have the read authority of RAID equipment;
(2) Formatting RAID1 equipment into an XFS file system, and mounting the XFS file system to a system directory of a controller;
(3) And storing the database file under the XFS file system directory, starting the database service, and enabling the main controller to read and write normally and enabling other alternative controllers to read the database data normally.
4. In response to receiving a request for database update, the host controller database service writes data into the database file while updating its bitmap data changes. Mainly comprises the following steps:
(1) When the main controller receives a request for updating the database, namely a write request, the main controller database service writes data into a database file;
(2) And updating bitmap data change of bitmap00X corresponding to the main controller, and recording the writing request.
5. In response to a failure of any one member disk of the RAID1 device, the other member disk may still normally service the database. Mainly comprises the following steps:
(1) When any member disk in the RAID1 equipment fails, the member disk is set to a fail state (failure state);
(2) The other member disk in the RAID1 device has full backup data of the failed hard disk and can still continue to service the database.
Responding to the failure of the main controller, taking over the read-write permission of the RAID1 device by the controller, copying the bitmap data of the main controller to the bitmap space of the main controller, and continuing to provide database service.
Mainly comprises the following steps:
(1) The takeover controller obtains the read-write permission of the RAID1 device, copies the bitmap data of the main controller into the bitmap space corresponding to the main controller, and empties the bitmap data of the main controller;
(2) And the takeover controller starts RAID1 data synchronization according to the bitmap data, recovers the data which is not written in when the main controller fails, executes the data synchronization, and ensures the consistency of the data.
(3) The takeover controller acquires the read-write permission of the RAID1 equipment and continues to provide the database service.
The data synchronization device, the management device, the controller, the data synchronization system and the computer readable storage medium provided in the embodiments of the present invention are described below, and the data synchronization device, the management device, the controller, the data synchronization system and the computer readable storage medium described below and the data synchronization method described above may be referred to correspondingly.
Referring to fig. 5, fig. 5 is a block diagram of a data synchronization device according to an embodiment of the present invention, where the device is applied to a management apparatus, and may include:
a storage device setting module 501, configured to set a database and a bitmap corresponding to each controller in a storage device;
the main controller setting module 502 is configured to set the main controller, so that the main controller obtains the read-write permission of the database separately, and updates the data writing position to the bitmap of the main controller when executing the data writing task to the database;
The takeover controller setting module 503 is configured to set the takeover controller when it is determined that the host controller fails, so that the takeover controller takes over the read-write permission of the database, and synchronize data in the bitmap of the host controller to the bitmap of the takeover controller, so as to resume execution of the data write service that is not completed by the host controller according to the bitmap of the takeover controller.
Optionally, the apparatus may further include:
the disk array assembly module is used for selecting a plurality of hard disk devices in the storage system and utilizing the plurality of hard disk devices to build a disk array so that the disk array at least allows one hard disk device to be damaged without interrupting storage service; the disk array is set as a storage device.
Optionally, the disk array assembly module is specifically configured to:
dividing partition spaces with preset sizes in each hard disk device;
and constructing a disk array by utilizing the partition space corresponding to each hard disk device.
Optionally, the takeover controller setting module 503 may further include:
the data writing service issuing module is used for issuing the unfinished data writing service of the main controller to the takeover controller so that the takeover controller can determine the completion progress of the main controller on the data writing service according to the data writing service and the bitmap of the takeover controller after synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, and resume executing the unfinished data writing service of the main controller according to the completion progress.
Optionally, the apparatus may further include:
the mapping relation setting module is used for dividing a storage space in the storage device into a plurality of data blocks and setting a one-to-one mapping relation between each bit in the bitmap and the data blocks;
and the mapping relation issuing module is used for issuing the mapping relation information recorded with the mapping relation to each controller so that the main controller updates the corresponding bit in the bitmap of the main controller according to the data block corresponding to the data writing position.
Referring to fig. 6, fig. 6 is a block diagram of another data synchronization apparatus according to an embodiment of the present invention, where the apparatus is applied to a controller, and may include:
a main controller module 601, configured to, when the managed device is set as a main controller, separately obtain a read-write permission of a database in the storage device, and update a data writing location to a bitmap corresponding to itself in the storage device when performing a data writing task to the database; the storage equipment is provided with bitmaps which are in one-to-one correspondence with the controllers;
and the take-over module 602 is configured to take over the read-write permission of the database when the managed device is set as the take-over controller, and synchronize the data in the bitmap of the main controller to the bitmap corresponding to the host, so as to resume executing the data writing service which is not completed by the main controller according to the bitmap corresponding to the host.
Optionally, the takeover module 602 may further include:
and the service recovery sub-module is used for determining the completion progress of the main controller on the data writing service according to the data writing service and the bitmap corresponding to the main controller when the data writing service which is issued by the management equipment and is not completed by the main controller is received and the data in the bitmap of the main controller is synchronized to the bitmap corresponding to the main controller, and recovering and executing the data writing service which is not completed by the main controller according to the completion progress.
Optionally, the apparatus may further include:
and the clearing module is used for clearing the data in the bitmap of the main controller.
Alternatively, the main controller module 601 may be configured to:
obtaining mapping relation information; the mapping relation information records a one-to-one mapping relation between each bit in the bitmap and each data block in the storage device; and updating the corresponding bit in the bitmap according to the data block corresponding to the data writing position.
Referring to fig. 7, fig. 7 is a block diagram illustrating a management apparatus according to an embodiment of the present invention, where the management apparatus 10 includes a processor 11 and a memory 12; wherein the memory 12 is used for storing a computer program; the processor 11 is configured to execute the data synchronization method provided in the foregoing embodiment when executing the computer program.
For the specific process of the data synchronization method, reference may be made to the corresponding content provided in the foregoing embodiment, and no further description is given here.
The memory 12 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the storage mode may be transient storage or permanent storage.
In addition, the management device 10 further includes a power supply 13, a communication interface 14, an input-output interface 15, and a communication bus 16; wherein, the power supply 13 is used for providing working voltage for each hardware device on the management device 10; the communication interface 14 can create a data transmission channel between the management device 10 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present invention, which is not specifically limited herein; the input/output interface 15 is used for obtaining external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
Referring to fig. 8, fig. 8 is a block diagram of a controller according to an embodiment of the present invention, and a controller 20 according to an embodiment of the present invention includes a processor 21 and a memory 22; wherein the memory 22 is used for storing a computer program; the processor 21 is configured to execute the data synchronization method provided in the foregoing embodiment when executing the computer program.
For the specific process of the data synchronization method, reference may be made to the corresponding content provided in the foregoing embodiment, and no further description is given here.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk or an optical disk, and the storage mode may be transient storage or permanent storage.
In addition, the controller 20 further includes a power supply 23, a communication interface 24, an input-output interface 25, and a communication bus 26; wherein, the power supply 23 is used for providing working voltage for each hardware device on the controller 20; the communication interface 24 can create a data transmission channel between the controller 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present invention, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
Referring to fig. 1, fig. 1 is a block diagram of a data synchronization system according to an embodiment of the present invention, where the apparatus may include:
A storage device 30 for setting a database and a bitmap corresponding to each controller one by one;
a management device 10 for executing the data synchronization method applied to the management device as described above;
and a controller 20 for performing the data synchronization method applied to the controller as described above.
Since the embodiments of the system portion and the embodiments of the data synchronization method portion correspond to each other, the embodiments of the storage medium portion are referred to for description of the embodiments of the data synchronization method portion, and are not repeated herein.
The embodiment of the invention also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the steps of the data synchronization method of any embodiment are realized.
Since the embodiments of the computer readable storage medium portion and the embodiments of the data synchronization method portion correspond to each other, the embodiments of the storage medium portion are referred to the description of the embodiments of the data synchronization method portion, and are not repeated herein.
In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data synchronization method, the device, the management equipment, the controller, the system and the medium provided by the invention are described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Claims (15)
1. A data synchronization method, applied to a management device, comprising:
setting a database and bitmaps corresponding to the controllers one by one in the storage equipment;
setting a main controller to enable the main controller to independently obtain the read-write permission of the database, and updating a data writing position into a bitmap of the main controller when executing a data writing task to the database;
and setting a takeover controller when the main controller is determined to be faulty, so that the takeover controller takes over the read-write permission of the database, and synchronizing data in the bitmap of the main controller to the bitmap of the takeover controller, so that the data writing service which is not completed by the main controller is restored to be executed according to the bitmap of the takeover controller.
2. The data synchronization method according to claim 1, further comprising, before setting the database and the bitmap corresponding to each controller one to one in the storage device:
selecting a plurality of hard disk devices in a storage system, and constructing a disk array by utilizing the plurality of hard disk devices, so that the disk array at least allows one hard disk device to be damaged without interrupting storage service;
And setting the disk array as the storage device.
3. The method of claim 2, wherein the constructing a disk array using the plurality of hard disk devices comprises:
partitioning a partition space with a preset size in each hard disk device;
and constructing the disk array by utilizing the partition space corresponding to each hard disk device.
4. The data synchronization method according to claim 1, further comprising, after setting up the takeover controller:
and transmitting the unfinished data writing service of the main controller to the takeover controller, so that the takeover controller can determine the completion progress of the main controller on the data writing service according to the data writing service and the bitmap of the takeover controller after synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, and resume executing the unfinished data writing service of the main controller according to the completion progress.
5. The data synchronization method according to any one of claims 1 to 4, further comprising:
dividing a storage space in the storage device into a plurality of data blocks, and setting a one-to-one mapping relation between each bit in the bitmap and the data blocks;
And transmitting the mapping relation information recorded with the mapping relation to each controller so that the main controller updates the corresponding bit in the bitmap of the main controller according to the data block corresponding to the data writing position.
6. A method of data synchronization, for use with a controller, the method comprising:
when the managed device is set as a main controller, the read-write permission of a database in the storage device is independently obtained, and when a data writing task is executed to the database, a data writing position is updated to a bitmap corresponding to the managed device in the storage device; the storage equipment is provided with bitmaps which are in one-to-one correspondence with the controllers;
when the management device is set as a takeover controller, taking over the read-write permission of the database, and synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the data, so that the data writing service which is not completed by the main controller is restored to be executed according to the bitmap corresponding to the data writing service.
7. The data synchronization method according to claim 6, further comprising, after being set as a takeover controller by the management device:
when the unfinished data writing service of the main controller issued by the management equipment is received and the synchronization of the data in the bitmap of the main controller to the bitmap corresponding to the main controller is completed, determining the completion progress of the main controller on the data writing service according to the data writing service and the bitmap corresponding to the main controller, and recovering and executing the unfinished data writing service of the main controller according to the completion progress.
8. The data synchronization method according to claim 6, further comprising, after synchronizing data in the bitmap of the main controller to its own corresponding bitmap:
and clearing the data in the bitmap of the main controller.
9. The method according to any one of claims 6 to 8, wherein updating the data writing location to the bitmap corresponding to itself in the storage device includes:
obtaining mapping relation information; the mapping relation information records a one-to-one mapping relation between each bit in the bitmap and each data block in the storage device;
and updating the corresponding bit in the bitmap according to the data block corresponding to the data writing position.
10. A data synchronization device, characterized by being applied to a management apparatus, comprising:
the storage device setting module is used for setting a database and bitmaps corresponding to the controllers one by one in the storage device;
the main controller setting module is used for setting a main controller so that the main controller can independently obtain the read-write permission of the database, and when a data writing task is executed to the database, the data writing position is updated to the bitmap of the main controller;
And the takeover controller setting module is used for setting a takeover controller when the main controller is determined to be faulty so that the takeover controller takes over the read-write permission of the database, and synchronizing the data in the bitmap of the main controller to the bitmap of the takeover controller, so that the unfinished data writing service of the main controller is restored to be executed according to the bitmap of the takeover controller.
11. A data synchronization device, for use in a controller, comprising:
the main controller module is used for independently obtaining the read-write permission of a database in the storage device when the managed device is set as the main controller, and updating the data writing position to a bitmap corresponding to the main controller in the storage device when a data writing task is executed to the database; the storage equipment is provided with bitmaps which are in one-to-one correspondence with the controllers;
and the take-over module is used for taking over the read-write permission of the database and synchronizing the data in the bitmap of the main controller to the bitmap corresponding to the main controller when the management equipment is set as the take-over controller so as to resume executing the unfinished data writing service of the main controller according to the bitmap corresponding to the main controller.
12. A management apparatus, characterized by comprising:
a memory for storing a computer program;
processor for implementing the data synchronization method according to any one of claims 1 to 5 when executing said computer program.
13. A controller, comprising:
a memory for storing a computer program;
processor for implementing a data synchronization method according to any of claims 6 to 9 when executing said computer program.
14. A data synchronization system, comprising:
the storage device is used for setting a database and bitmaps corresponding to the controllers one by one;
a management device for performing the data synchronization method according to any one of claims 1 to 5;
a controller for performing the data synchronization method according to any one of claims 6 to 9.
15. A computer readable storage medium having stored therein computer executable instructions which when loaded and executed by a processor implement the data synchronization method of any one of claims 1 to 5 and/or the data synchronization method of any one of claims 6 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311329650.6A CN117349381A (en) | 2023-10-13 | 2023-10-13 | Data synchronization method, device, management equipment, controller, system and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311329650.6A CN117349381A (en) | 2023-10-13 | 2023-10-13 | Data synchronization method, device, management equipment, controller, system and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117349381A true CN117349381A (en) | 2024-01-05 |
Family
ID=89356991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311329650.6A Pending CN117349381A (en) | 2023-10-13 | 2023-10-13 | Data synchronization method, device, management equipment, controller, system and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117349381A (en) |
-
2023
- 2023-10-13 CN CN202311329650.6A patent/CN117349381A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8504765B2 (en) | Method of improving efficiency of capacity of volume used for copy function and apparatus thereof | |
| US6950915B2 (en) | Data storage subsystem | |
| US6990611B2 (en) | Recovering data from arrays of storage devices after certain failures | |
| JP4809040B2 (en) | Storage apparatus and snapshot restore method | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| US20040103104A1 (en) | Snapshot creating method and apparatus | |
| US8255649B2 (en) | Remote copy control method and system in storage cluster environment | |
| CN115167782B (en) | Temporary storage copy management method, system, equipment and storage medium | |
| JP2007018312A (en) | Disk array device and its control method | |
| CN102025758B (en) | Method, device and system for recovering data copy in distributed system | |
| CN113885809B (en) | Data management system and method | |
| CN105740049A (en) | Control method and apparatus | |
| CN117763046A (en) | Method, device, equipment and storage medium for synchronizing data among clusters | |
| CN116204137B (en) | Distributed storage system, control method, device and equipment based on DPU | |
| CN111381770B (en) | Data storage switching method, device, equipment and storage medium | |
| CN111176886A (en) | Database mode switching method and device and electronic equipment | |
| CN117349381A (en) | Data synchronization method, device, management equipment, controller, system and medium | |
| CN111400098B (en) | Copy management method and device, electronic equipment and storage medium | |
| CN113703673A (en) | Single-computer data storage method and related device | |
| CN109271109B (en) | Data storage method and HBA card | |
| JP2021174392A (en) | Remote copy system and remote copy control method | |
| CN105844178B (en) | JBOD mass storage data security method | |
| US20240036989A1 (en) | Storage system and management method | |
| CN113467717B (en) | Dual-machine volume mirror image management method, device and equipment and readable storage medium | |
| CN117608494B (en) | Storage method and system of cloud computing cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |