CN116821092A - Data migration method, device and storage medium based on MasterHA - Google Patents
Data migration method, device and storage medium based on MasterHA Download PDFInfo
- Publication number
- CN116821092A CN116821092A CN202310728215.4A CN202310728215A CN116821092A CN 116821092 A CN116821092 A CN 116821092A CN 202310728215 A CN202310728215 A CN 202310728215A CN 116821092 A CN116821092 A CN 116821092A
- Authority
- CN
- China
- Prior art keywords
- library
- slave
- masterha
- master library
- new master
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 35
- 230000005012 migration Effects 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000004590 computer program Methods 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 2
- 238000010367 cloning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a fault migration method, a device and a storage medium based on a MasterHA, wherein a RBD device in Ceph is adopted as a storage device in a database of the MasterHA, a master library is detected at regular time through a management node of the MasterHA, a preparation new master library is selected from a slave library again when the master library is down, a relay log difference of the preparation new master library and the slave library is applied to the slave library, the preparation new master library is used as a final new master library, a master-slave relationship of the final new master library and the slave library is established, the final new master library is monitored in real time by the slave library, and binary log event changes of the final new master library are copied to respective relay logs and are applied to the respective databases, so that data migration is realized. Therefore, the invention reduces the possibility of service data loss and supports online capacity expansion by utilizing the high availability of RBD equipment, and applies the relay log difference between the final new master library and all slave libraries to all slave libraries, thereby ensuring the consistency of the data of the master library and the slave libraries after migration.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data migration method and apparatus based on MasterHA, and a storage medium.
Background
With the development of information technology, enterprises increasingly rely on informatization management, data information of each business application is mainly stored in a database, and the requirements of the enterprises on continuity of access to the data are higher and higher, so that the high availability of the database is becoming a great importance of informatization construction of the enterprises in order to avoid various losses caused by interruption of the data.
At present, most enterprises generally adopt a mysql database, but the mysql database has the following defects that when a disk of a mysql node fails, service data is lost, when data migration is required to be carried out due to node failure, the situation of inconsistent data after migration often occurs, and online capacity expansion cannot be carried out on the disk of the mysql node, so that the current mysql database has low availability.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the invention provides a data migration method, a data migration device and a storage medium based on a MasterHA, which improve the availability of a database.
In order to solve the technical problems, the invention adopts the following technical scheme:
in a first aspect, the present invention provides a fault migration method based on MasterHA, including:
detecting a main library at regular time through a management node of a MasterHA, wherein the database of the MasterHA adopts RBD equipment in Ceph as storage equipment;
when the master library is down, marking the master library as a down master library, selecting a preparation new master library from all slave libraries, calculating the relay log difference between the preparation new master library and all slave libraries, applying the relay log difference to all slave libraries, and taking the preparation new master library as a final new master library;
and establishing a master-slave relation between the final new master library and all the slave libraries, so that all the slave libraries monitor the final new master library in real time, and when the change of the binary log event of the final new master library is found, copying the change to respective relay logs and applying the relay logs to respective databases to complete data migration.
The invention has the advantages that the database of the MasterHA adopts RBD equipment in the Ceph as storage equipment so as to realize the connection of the Ceph and the MasterHA, the high availability of the RBD equipment is utilized to reduce the possibility of losing service data, simultaneously, the online capacity expansion is supported, the management node detects the main database at regular time, when the main database is down, the final new main database is reelected, the relay log difference of the final new main database and all the slave databases is applied to all the slave databases, the master-slave relation of the final new main database and all the slave databases is reestablished, the slave databases can monitor the final new main database in real time and copy the binary log event change of the final new main database to respective relay logs and apply the binary log event change to the respective databases, thereby ensuring the consistency of the transferred main database and the slave databases and improving the availability of the databases.
Optionally, the detecting the master library by the management node at regular time includes:
when the management node detects the main library, the main library responds to the detection and sends the response to the management node;
and if the management node does not receive the response, re-detecting the main library until the re-detection times exceed a threshold value, stopping re-detection, and marking the main library as a down main library.
According to the description, when the number of times of re-detecting the main library by the management node exceeds the threshold, the detection is stopped and the main library is marked as the down main library, so that erroneous judgment is avoided.
Optionally, the selecting, by the management node, a new master library from all slave libraries includes:
all the slave libraries read binary log events from the master library in real time, copy the binary log events into respective relay logs and finish the update of the slave libraries;
the management node takes the latest updated slave library as a preparation new master library.
According to the above description, the slave library keeps consistent with the master library in real time, and the latest updated slave library is used as a preparation new master library, so that the integrity and the comprehensiveness of data are ensured.
Optionally, the completing the slave library update includes:
and all the slave libraries read the corresponding relay logs through the SQL threads, and apply the relay logs to the databases to finish the update of the slave libraries.
From the above description, the slave library will apply the relay log to the respective database through the SQL thread, i.e. execute the relay log to the respective database, so as to achieve that the data of the slave library and the master library are consistent.
Optionally, the establishing a master-slave relationship between the final new master library and all the slave libraries further includes:
and removing the information of the down host library from the project configuration file.
According to the description, the information of the down main library is removed from the project configuration file, so that the project configuration file is updated, and the instantaneity of the project configuration file is ensured.
Optionally, the database of the MasterHA adopts an RBD device in Ceph as a storage device further includes:
and monitoring the disk state of the database in real time through the Ceph.
According to the description, the disk state of the database can be monitored in real time, so that fine granularity management of the database is realized.
Optionally, the threshold is 3.
In a second aspect, a data migration apparatus based on MasterHA is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements a data migration method based on MasterHA according to the first aspect when executing the computer program.
In a third aspect, there is provided a computer readable storage medium having a computer program stored thereon, the computer program, when executed, implementing a MasterHA-based data migration method according to the first aspect.
Wherein, the data migration device based on MasterHA provided in the second aspect and the computer readable storage medium provided in the third aspect correspond to technical effects with reference to the related description of the data migration method based on MasterHA provided in the first aspect.
Drawings
Fig. 1 is a flowchart of a data migration method based on MasterHA according to an embodiment of the present invention;
fig. 2 is an overall flow diagram of a data migration method based on MasterHA according to an embodiment of the present invention;
fig. 3 is an overall framework diagram of a data migration method based on MasterHA according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a secondary repository copying and applying binary log event changes of a final new primary repository to respective relay logs to respective databases in accordance with an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data migration device based on MasterHA according to an embodiment of the present invention.
[ reference numerals description ]
1. A data migration device based on a MasterHA;
2. a processor;
3. a memory.
Detailed Description
In order that the above-described aspects may be better understood, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
Example 1
Referring to fig. 1 to 4, the present invention provides a data migration method based on MasterHA, comprising the steps of:
s1, detecting a main library at regular time through a management node of a MasterHA, wherein the database of the MasterHA adopts RBD equipment in Ceph as storage equipment;
in this embodiment, as shown in fig. 3, the master ha (Master High Availability, high availability configuration and failover) management node detects the master library at a timing set to detect the master library every second, i.e. every second by the management node, where the master library supports read-write operations, all read-write operations are recorded in the binary log in an "event" manner, and Ceph (distributed storage system) supports online capacity expansion, snapshot, cloning, and other operations on RBD (RADOS block device, block device service) databases.
At this time, the detecting the master library by the management node in step S1 includes:
s11, when the management node detects the main library, the main library responds to the detection and sends the response to the management node;
and S12, if the management node does not receive the response, re-detecting the main library, stopping re-detection until the re-detection times exceed a threshold value, and marking the main library as a down main library.
In this embodiment, the threshold is set to 3 at this time, that is, when the number of times the management node re-detects the main library exceeds 3, the management node will stop re-detecting and mark the main library as the down main library.
At this time, the database of the MasterHA in step S1 adopts the RBD device in Ceph as the storage device further includes:
s13, monitoring the disk state of the database in real time through the Ceph.
In this embodiment, the Ceph monitors the disk state of the database in real time, and the instrument panel of the Ceph is matched with the promethaus (open source monitoring system) and the Grafana (visualization tool) to perform fine monitoring and alarming, so as to realize effective management.
S2, when the master library is down, marking the master library as the down master library, selecting a preparation new master library from all slave libraries, calculating the relay log difference between the preparation new master library and all slave libraries, applying the relay log difference to all slave libraries, and taking the preparation new master library as a final new master library;
in this embodiment, as shown in FIG. 4, the slave library supports only read operations and maintains communication with the master server through I/O threads.
At this time, the step S2 of the management node electing a new master library from all slave libraries includes:
s21, all the slave libraries read binary log events from the master library in real time, and copy the binary log events to respective relay logs to finish the update of the slave library;
in this embodiment, as shown in fig. 4, the slave library reads the binary log event from the master library in real time through the I/O thread and copies the binary log event to the respective relay log.
At this time, the completion of the slave library update in step S21 includes:
s211, all the slave libraries read the corresponding relay logs through the SQL threads, and apply the relay logs to the databases to finish the update of the slave libraries.
S22, the management node takes the latest updated slave library as a preparation new master library.
And S3, establishing a master-slave relation between the final new master library and all the slave libraries, so that all the slave libraries monitor the final new master library in real time, and copying the changes to respective relay logs and applying the changes to respective databases when the changes of the binary log events of the final new master library are found, thereby completing data migration.
In this embodiment, all the slave libraries monitor the final new master library in real time, when the change of the binary log event of the final new master library is found, the change of the binary log event is copied to respective relay logs through the I/O thread, the respective corresponding relay logs are read through the respective SQL threads, and the relay logs are applied to the respective databases to complete data migration.
At this time, the establishing a master-slave relationship between the final new master library and all the slave libraries in step S3 further includes:
s31, removing the information of the down master library from the project configuration file.
Example two
Referring to fig. 5, a MasterHA-based data migration apparatus 1 includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2, wherein the processor 2 implements the steps of the first embodiment when executing the computer program.
Example III
The present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed, implements the steps of the first embodiment.
Since the system/device described in the foregoing embodiments of the present invention is a system/device used for implementing the method of the foregoing embodiments of the present invention, those skilled in the art will be able to understand the specific structure and modification of the system/device based on the method of the foregoing embodiments of the present invention, and thus will not be described in detail herein. All systems/devices used in the methods of the above embodiments of the present invention are within the scope of the present invention.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions.
It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the terms first, second, third, etc. are for convenience of description only and do not denote any order. These terms may be understood as part of the component name.
Furthermore, it should be noted that in the description of the present specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with the embodiment or example being included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art upon learning the basic inventive concepts. Therefore, the appended claims should be construed to include preferred embodiments and all such variations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, the present invention should also include such modifications and variations provided that they come within the scope of the following claims and their equivalents.
Claims (9)
1. A MasterHA-based data migration method, comprising:
detecting a main library at regular time through a management node of a MasterHA, wherein the database of the MasterHA adopts RBD equipment in Ceph as storage equipment;
when the master library is down, marking the master library as a down master library, selecting a preparation new master library from all slave libraries, calculating the relay log difference between the preparation new master library and all slave libraries, applying the relay log difference to all slave libraries, and taking the preparation new master library as a final new master library;
and establishing a master-slave relation between the final new master library and all the slave libraries, so that all the slave libraries monitor the final new master library in real time, and when the change of the binary log event of the final new master library is found, copying the change to respective relay logs and applying the relay logs to respective databases to complete data migration.
2. The MasterHA-based data migration method of claim 1 wherein said managing node timing detection of the master library comprises:
when the management node detects the main library, the main library responds to the detection and sends the response to the management node;
and if the management node does not receive the response, re-detecting the main library until the re-detection times exceed a threshold value, stopping re-detection, and marking the main library as a down main library.
3. The MasterHA-based data migration method of claim 1, wherein said management node electing a preliminary new master library from all slave libraries comprises:
all the slave libraries read binary log events from the master library in real time, copy the binary log events into respective relay logs and finish the update of the slave libraries;
the management node takes the latest updated slave library as a preparation new master library.
4. A MasterHA-based data migration method as claimed in claim 3, wherein said completing the slave library update comprises:
and all the slave libraries read the corresponding relay logs through the SQL threads, and apply the relay logs to the databases to finish the update of the slave libraries.
5. A MasterHA-based data migration method as claimed in claim 1, and wherein said establishing a master-slave relationship between said final new master library and all said slave libraries further comprises:
and removing the information of the down host library from the project configuration file.
6. The MasterHA-based data migration method of claim 1, wherein the MasterHA database employs RBD devices in Ceph as storage devices further comprises:
and monitoring the disk state of the database in real time through the Ceph.
7. A MasterHA-based data migration method as claimed in claim 2, characterized in that said threshold value is 3.
8. A MasterHA-based data migration apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
9. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed, implements the method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310728215.4A CN116821092A (en) | 2023-06-19 | 2023-06-19 | Data migration method, device and storage medium based on MasterHA |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310728215.4A CN116821092A (en) | 2023-06-19 | 2023-06-19 | Data migration method, device and storage medium based on MasterHA |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116821092A true CN116821092A (en) | 2023-09-29 |
Family
ID=88112060
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310728215.4A Pending CN116821092A (en) | 2023-06-19 | 2023-06-19 | Data migration method, device and storage medium based on MasterHA |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116821092A (en) |
-
2023
- 2023-06-19 CN CN202310728215.4A patent/CN116821092A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291787B (en) | Main and standby database switching method and device | |
| US11256715B2 (en) | Data backup method and apparatus | |
| US9189348B2 (en) | High availability database management system and database management method using same | |
| US9189338B2 (en) | Disaster recovery failback | |
| CN111078667B (en) | Data migration method and related device | |
| US7457830B1 (en) | Method and system of replicating data using a recovery data change log | |
| JP2007164769A (en) | Method, system and program for managing failure in mirrored system (management of failure in mirrored system) | |
| US10114703B2 (en) | Flash copy for disaster recovery (DR) testing | |
| CN105049258B (en) | The data transmission method of network disaster tolerance system | |
| CN104615511A (en) | Host batch recovery processing method and device based on double centers | |
| CN114064374A (en) | Fault detection method and system based on distributed block storage | |
| CN110121694A (en) | A kind of blog management method, server and Database Systems | |
| CN111046051A (en) | Data consistency implementation method, device and storage medium | |
| JP2006185108A (en) | Management computer for managing data of storage system, and data management method | |
| CN116361078A (en) | Data synchronization method, device, system and medium | |
| CN116821092A (en) | Data migration method, device and storage medium based on MasterHA | |
| CN111400098B (en) | Copy management method and device, electronic equipment and storage medium | |
| CN110121712A (en) | A kind of blog management method, server and Database Systems | |
| CN113986450A (en) | Virtual machine backup method and device | |
| US9471409B2 (en) | Processing of PDSE extended sharing violations among sysplexes with a shared DASD | |
| CN112433887A (en) | Database data synchronization method, system, device and storage medium | |
| US11599428B2 (en) | Protecting databases in a distributed availability group | |
| US11232131B1 (en) | Synchronous block level replication across availability zones | |
| CN111198783B (en) | Data access method, device, system, equipment and storage medium | |
| CN111813786B (en) | Defect detection/processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |