CN105955989B - Method for establishing master server and slave server of cloud platform database - Google Patents
Method for establishing master server and slave server of cloud platform database Download PDFInfo
- Publication number
- CN105955989B CN105955989B CN201610244979.6A CN201610244979A CN105955989B CN 105955989 B CN105955989 B CN 105955989B CN 201610244979 A CN201610244979 A CN 201610244979A CN 105955989 B CN105955989 B CN 105955989B
- Authority
- CN
- China
- Prior art keywords
- server
- slave
- master
- master server
- slave server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The invention provides a method for establishing a master server and a slave server of a cloud platform database, which comprises the following steps: s1, selecting or creating a main server, performing customized configuration on the main server, and then performing snapshot operation; s2, creating a slave server data disc by the snapshot in the step S1, and starting the slave server; s3, only recording the binlog file name and the position information corresponding to the server after the database service is successfully started for the first time; s4, reading CRC32 check values of corresponding database operation records in the binlog file of the master server and the slave server respectively according to the binlog file name and the position information in the step S3, judging, and if the check values are the same, starting the synchronous operation of the master server and the slave server; if not, the synchronization operation of the master server and the slave server is not started, and the steps S1 to S3 are repeatedly executed until the master server and the slave server are successfully established. The invention can effectively reduce the occupied space of the disk, quickly establish the master server and the slave server under the condition of not blocking the service of the master server, and ensure the data consistency of the master server and the slave server.
Description
Technical Field
The invention relates to the technical field of cloud computing, in particular to a method for establishing a master server and a slave server of a cloud platform database.
Background
In the cloud platform, the database provides data storage and calling for the platform to realize various references such as virtual computing, virtual storage, development environment and the like. The Mysql database is a database architecture commonly used in the current cloud platform, and the built-in copy function of the Mysql database is the basis for constructing large and high-performance application programs. The data of Mysql is distributed to a plurality of systems, and the distribution mechanism is realized by copying the data of one host of Mysql to other hosts (slaves) and performing the process again. One server acts as a master and one or more other servers act as slaves during replication. The primary server writes updates to a binary log (binlog) file and maintains an index of the file to track log cycles. These logs may record updates sent to the slave server. When a slave server connects to the master server, it informs the master server of the last successfully updated location the slave server read in the log. The slave receives any updates that have occurred since then, and blocks and waits for the master to notify of the new updates.
The master-slave architecture of the Mysql database realizes the master-slave copying and read-write separation functions of the database, on one hand, the high availability of the system is ensured, and on the other hand, the concurrent load capacity of the database is improved. The above implementation needs to ensure data consistency and transparency requirements for users.
The key points for realizing the method for establishing the Mysql database master server and the Mysql database slave server are as follows: the slave server needs to have data from a master-slave server data snapshot point to ensure that it is synchronized from the snapshot point master server. Currently, methods such as creating a snapshot point for a master server by using a MySQL-based software tool are used to copy a data snapshot to a slave server, thereby realizing synchronization with the master server. The establishing method can ensure the consistency of data and ensure that the service of the database is not interrupted. However, when the data amount of the master server is large, the time for creating the slave server is proportional to the data amount of the master server, and a large amount of time is required for copying the data to the slave server; in addition, since the data of the master server and each slave server are the same, the disk space is wasted to a certain extent.
In view of this, there is a need to improve a method for establishing a master server and a slave server of a Mysql database of a cloud platform in the prior art, so as to solve the above problems.
Disclosure of Invention
The invention aims to disclose a method for establishing a master server and a slave server of a cloud platform database, which is used for reducing the time consumption of creating the slave servers and reducing the waste of disk space on the premise of ensuring the consistency of data in the master server and the slave servers and not blocking the service of the master server.
In order to achieve the above object, the present invention provides a method for establishing a master server and a slave server of a cloud platform database, wherein the method comprises:
s1, selecting or creating a main server, carrying out customized configuration on the main server, and carrying out snapshot operation on the main server;
s2, creating a slave server data disc according to the snapshot in the step S1, and starting the slave server;
s3, only recording the binlog file name and the position information corresponding to the server after the database service is successfully started for the first time;
s4, reading CRC32 check values of corresponding database operation records in the binlog files of the master server and the slave server respectively according to the binlog file name and the position information in the step S3, judging, and if the check values are the same, starting synchronous operation of the master server and the slave server; if not, the synchronization operation between the master server and the slave server is not started, and the steps S1 to S3 are repeatedly executed until the master server and the slave server are successfully established.
As a further improvement of the present invention, the customized configuration in step S1 includes a global variable configuration, a cache flush configuration, a distributed transaction configuration, and a master-slave replication event check configuration.
As a further refinement of the present invention, the data in the primary server data disk is created in step S2 with a distributed storage system that supports snapshots.
As a further improvement of the present invention, the distributed storage system comprises a Ceph distributed block storage system.
As a further improvement of the present invention, step S4 further includes a step of deleting the data disk of the master server and not opening the synchronization operation of the master server and the slave server when the CRC32 check values of the corresponding database operation records in the binlog files of the master server and the slave server are read and judged to be different according to the binlog file name and the position information in step S3, respectively.
As a further improvement of the invention, the cloud platform database is a MySQL database.
Compared with the prior art, the invention has the beneficial effects that: in the invention, the data in the created data disk of the main server is stored in a snapshot mode, so that the rapid copying of the data is realized on the disk level, and the occupation of the disk is solved; meanwhile, CRC32 check values of corresponding database operation records in the binlog files of the master server and the slave server are respectively read according to the binlog file name and the position information and judged, the master server and the slave server are established only when the judgment is consistent, the master server and the slave server are quickly established under the condition that the service of the master server is not blocked, and the consistency of data in the master server and the slave server is ensured.
Detailed Description
The present invention is described in detail below with reference to various embodiments, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should be able to make modifications and substitutions on the functions, methods, or structures of these embodiments without departing from the scope of the present invention.
In this embodiment, a method for establishing a master server and a slave server of a cloud platform database includes the following steps:
step S1, selecting or creating a master server and performing customized configuration on the master server, and performing snapshot operation on the master server. The customized configuration in step S1 includes a global variable configuration, a cache flush configuration, a distributed transaction configuration, and a master-slave replication event check configuration. Specifically, the operation code of the customized configuration is as follows:
sync_binlog＝1
innodb_flush_log_at_trx_commit＝1
innodb_support_xa＝1
binlog_checksum＝crc32。
the principle introduction of MySQL replication is as follows: the MySQL master library writes binlog at transaction commit and controls binlog flush to disk "floor" by sync _ binlog parameter. The standby library draws the binlog from the main library through the IO thread and records the binlog into a local relay log; and the local SQL thread applies the data in the relay log to a local database.
MySQL provides a sync _ binlog parameter to control the binlog of the database to be printed onto disk. Although binlog also has a binlog cache, MySQL has no relevant consideration for controlling the synchronization of the binlog cache to the file system cache. So we do not refer here to binlog cache.
By default, sync _ binlog is 0, meaning MySQL does not control the refreshing of binlog, and the file system itself controls the refreshing of its cache.
If sync _ binlog >0, indicating that every sync _ binlog transaction commits, the flush operation of the MySQL call file system flushes the cache. The safest is that sync _ binlog is 1, meaning that every time a transaction commits, MySQL flushes binlog. In this way, the system may lose 1 transaction of data in the event that the host operating system on which the database resides is corrupted or suddenly loses power. However, binlog is sequential IO, but sync _ binlog is set to 1, and multiple transactions are submitted simultaneously, which also greatly affects MySQL and IO performance. Although the patch mitigation by group commit can be achieved, the IO is affected by too high frequency of refresh.
Many MySQL DBA set sync _ binlog is not the safest 1 but 100 or 0. This sacrifices some consistency and allows for higher concurrency and performance.
In this embodiment, the cloud platform database is a MySQL database. The MySQL database supports unidirectional, asynchronous replication in which one server acts as a master and one or more other servers act as slaves. The master server writes updates to the binary log file and maintains an index of the log file to track log cycles. When a slave server connects to the master server, the slave server notifies the master server that the slave server read the last successfully updated location in the log. The slave receives any updates that have occurred since then, and blocks and waits for the master to notify the next update.
The master-slave server established by the configuration can ensure the consistency of data in the master-slave server or slave server after abnormal service, such as unexpected power failure restart, and the advantages of the master-slave server and the slave server are more obvious when a distributed storage system or a storage medium is located in a remote entity server.
And step S2, creating a slave server data disc according to the snapshot in the step S1, and starting the slave server after the customized configuration is carried out on the slave server data disc. The master server data disk is created with the distributed storage system supporting snapshots in step S2; preferably, the distributed storage system comprises a Ceph block storage system. The Ceph block storage system is a Linux PB-level distributed storage system, and the architecture of the Ceph block storage system can be roughly divided into four parts: client (data user), metadata server (caching and synchronizing distributed metadata), one object storage cluster (storing data and metadata as objects), cluster monitor (performing monitoring functions).
The data in the database is created and stored in a snapshot form, and the occupied space of a disk in the server can be solved. The server refers to a physical server, and the disk refers to a physical disk, a disk array (RAID), or a storage medium composed of a non-volatile storage element (NV-Memory).
And step S3, only recording the binlog file name and the position information corresponding to the first successful starting of the database service from the server.
Step S4, reading CRC32 check values of corresponding database operation records in the binlog files of the master server and the slave server respectively according to the binlog file name and the position information in the step S3, judging, and if the check values are the same, starting synchronous operation of the master server and the slave server; if not, the synchronization operation between the master server and the slave server is not started, and the steps S1 to S3 are repeatedly executed until the master server and the slave server are successfully established.
Preferably, the step S4 further includes a step of deleting the data disc of the master server and not opening the synchronous operation of the master server and the slave server when the CRC32 check values of the corresponding database operation records in the binlog files of the master server and the slave server are read and judged to be different according to the binlog file name and the position information in the step S3.
CRC32(Cyclic Redundancy Check) is a Cyclic Redundancy Check code. The CRC32 check has the advantages of extremely strong error detection capability and low cost, and is easy to realize by using an encoder and a detection circuit. From the viewpoint of its error detection capability, the probability of an error that it cannot detect is only 0.0047% or less. The method is far superior to parity check, arithmetic and check and the like in terms of performance and overhead. By adopting the method for checking the CRC32 check value, the consistency of data in the master server and the slave server can be ensured.
In the present embodiment, both the established master server and the slave server include an Agent program, and the Agent program may be responsible for operations such as binlog file synchronization configuration and MySQL status information query in the server.
The above-listed detailed description is only a specific description of a possible embodiment of the present invention, and they are not intended to limit the scope of the present invention, and equivalent embodiments or modifications made without departing from the technical spirit of the present invention should be included in the scope of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (3)
1. A method for establishing a master server and a slave server of a cloud platform database is characterized by comprising the following steps:
s1, selecting or creating a main server, carrying out customized configuration on the main server, and carrying out snapshot operation on the main server;
s2, creating a slave server data disc according to the snapshot in the step S1, and starting the slave server;
s3, only recording the binlog file name and the position information corresponding to the server after the database service is successfully started for the first time;
s4, reading CRC32 check values of corresponding database operation records in the binlog files of the master server and the slave server respectively according to the binlog file name and the position information in the step S3, judging, and if the check values are the same, starting synchronous operation of the master server and the slave server; if not, not starting the synchronous operation of the master server and the slave server, and repeatedly executing the steps S1 to S3 until the master server and the slave server are successfully established;
the established master server and the established slave server both comprise Agent programs;
the customized configuration in step S1 includes global variable configuration, cache refresh configuration, distributed transaction configuration, and master-slave replication event check configuration;
the cloud platform database is a MySQL database and supports unidirectional and asynchronous replication, one server serves as a master server in the replication process, one or more other servers serve as slave servers, the master server writes updates into a binary log file and maintains an index of the log file to track log circulation, when one slave server is connected to the master server, the slave server informs the master server of reading the last successfully updated position from the log, the slave server receives any updates occurring during the last successfully updated period, and then blocks and waits for the master server to inform the next update.
2. The method according to claim 1, wherein the master server data disk is created in a distributed storage system supporting snapshot in step S2.
3. The method of building of claim 2, wherein the distributed storage system comprises a Ceph distributed block storage system.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511034685 | 2015-12-31 | ||
| CN2015110346852 | 2015-12-31 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105955989A CN105955989A (en) | 2016-09-21 |
| CN105955989B true CN105955989B (en) | 2020-12-22 |
Family
ID=56917642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610244979.6A Active CN105955989B (en) | 2015-12-31 | 2016-04-19 | Method for establishing master server and slave server of cloud platform database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105955989B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804463B (en) * | 2017-05-03 | 2021-07-09 | 杭州海康威视数字技术股份有限公司 | Data synchronization method and device of MySQL database and electronic equipment |
| CN108763578B (en) * | 2018-06-07 | 2023-03-10 | 腾讯科技（深圳）有限公司 | Index file updating method and server |
| US11550953B2 (en) | 2020-09-16 | 2023-01-10 | Saudi Arabian Oil Company | Preserving cloud anonymity |
| CN112445780A (en) * | 2020-11-27 | 2021-03-05 | 上海上讯信息技术股份有限公司 | Master-slave architecture deployment method and device based on snapshot |
| CN114020850B (en) * | 2022-01-05 | 2022-04-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6247141B1 (en) * | 1998-09-24 | 2001-06-12 | Telefonaktiebolaget Lm Ericsson (Publ) | Protocol for providing replicated servers in a client-server system |
| CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
| CN103530290A (en) * | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
| CN104252502A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out data migration on database management platform |
| CN104506625A (en) * | 2014-12-22 | 2015-04-08 | 国云科技股份有限公司 | Method for improving reliability of metadata nodes of cloud databases |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9152686B2 (en) * | 2012-12-21 | 2015-10-06 | Zetta Inc. | Asynchronous replication correctness validation |
| CN103905220B (en) * | 2012-12-25 | 2018-02-27 | 腾讯科技（北京）有限公司 | Data synchronizing processing method and system |
| CN104252500B (en) * | 2013-06-29 | 2017-11-28 | 北京新媒传信科技有限公司 | The fault repairing method and device of a kind of database management platform |
| CN104657382B (en) * | 2013-11-21 | 2018-09-14 | 阿里巴巴集团控股有限公司 | Method and apparatus for MySQL principal and subordinate's server data consistency detections |
| CN104679614B (en) * | 2015-03-31 | 2018-03-27 | 成都文武信息技术有限公司 | A kind of Database Disaster Backup/Recover |
-
2016
- 2016-04-19 CN CN201610244979.6A patent/CN105955989B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6247141B1 (en) * | 1998-09-24 | 2001-06-12 | Telefonaktiebolaget Lm Ericsson (Publ) | Protocol for providing replicated servers in a client-server system |
| CN103530290A (en) * | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
| CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
| CN104252502A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out data migration on database management platform |
| CN104506625A (en) * | 2014-12-22 | 2015-04-08 | 国云科技股份有限公司 | Method for improving reliability of metadata nodes of cloud databases |
Non-Patent Citations (2)
| Title |
|---|
| 两种方法实现DNS主从服务器数据同步;徐阳等;《中国教育网络》;20090831;第76-77页 * |
| 基于数据同步的集群系统不间断服务的设计与实现;陈东江;《中国优秀硕士学位论文全文数据库信息科技辑》;20150515;I138-398 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105955989A (en) | 2016-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10664366B2 (en) | Third vote consensus in a cluster using shared storage devices | |
| US20200371884A1 (en) | Remote Data Replication Method and System | |
| US10909004B1 (en) | Online system checkpoint recovery orchestration | |
| US10235066B1 (en) | Journal destage relay for online system checkpoint creation | |
| CN105955989B (en) | Method for establishing master server and slave server of cloud platform database | |
| US10360120B2 (en) | High availability failover manager | |
| US10725878B2 (en) | Storage apparatus, storage system, and control method of storage system for dynamically securing free space when a storage apparatus is disused | |
| EP2391968B1 (en) | System and method for secure and reliable multi-cloud data replication | |
| KR101914019B1 (en) | Fast crash recovery for distributed database systems | |
| US8725951B2 (en) | Efficient flash memory-based object store | |
| US8904117B1 (en) | Non-shared write-back caches in a cluster environment | |
| US20080243954A1 (en) | Generating and indicating incremental backup copies from virtual copies of a data set | |
| US7761431B2 (en) | Consolidating session information for a cluster of sessions in a coupled session environment | |
| US10664358B1 (en) | Ensure volume consistency for online system checkpoint | |
| US20240045598A1 (en) | Cloud object storage and versioning system | |
| US10503620B1 (en) | Parity log with delta bitmap | |
| US20220019555A1 (en) | Snapshot and restoration of distributed file system | |
| US20190188097A1 (en) | Mirrored write ahead logs for data storage system | |
| US10095700B2 (en) | Persistent file handle object container memory expiry | |
| US8850139B2 (en) | Changing ownership of cartridges | |
| US10671488B2 (en) | Database in-memory protection system | |
| WO2024051027A1 (en) | Data configuration method and system for big data | |
| EP3794451A1 (en) | Parity log with by-pass | |
| JP2009265973A (en) | Data synchronization system, failure recovery method, and program | |
| CN111400098A (en) | Copy management method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu Applicant after: Huayun data holding group Co., Ltd Address before: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu Applicant before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |