CN105955989A - Method for establishing master and slave servers of cloud platform database - Google Patents
Method for establishing master and slave servers of cloud platform database Download PDFInfo
- Publication number
- CN105955989A CN105955989A CN201610244979.6A CN201610244979A CN105955989A CN 105955989 A CN105955989 A CN 105955989A CN 201610244979 A CN201610244979 A CN 201610244979A CN 105955989 A CN105955989 A CN 105955989A
- Authority
- CN
- China
- Prior art keywords
- server
- master
- binlog
- master server
- building
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention provides a method for establishing master and slave servers of a cloud platform database. The method comprises the steps that at the step S1, master and slave servers are selected or established, customized configuration is carried out to the servers, and a snapshot operation is implemented; at the step S2, a snapshot at the step S1 is used to establish a data disk of the slave server, and the slave server is started; at the step S3, a name and position information of a corresponding binlog document existing after a database service of the slave server is started successfully for the first time are recorded; and at the step S4, CRC32 verification values of database operation records corresponding to the binlog documents of the master and slave servers are read respectively according to the name and the position information of the binlog document obtained at the step S3, and the verification values are judged, wherein an synchronous operation of the master and slave servers is started if the verification values are same, and the synchronous operation of the master and slave servers are not started, and the steps from S1 to S3 are executed repeatedly till the master and slave servers are successfully established if the verification values are different. The method provided by the invention has the advantages that disk occupied space can be effectively reduced; the master and slave servers can be established rapidly without blocking in services of the master server; and consistency in data of the master and slave servers is ensured.
Description
Technical field
The present invention relates to field of cloud computer technology, particularly relate to a kind of cloud platform data base's principal and subordinate's server
Method for building up.
Background technology
In cloud platform, data base realizes virtual computing, virtual memory, development environment etc. for platform
Multiple offer data of quoting store and call.Mysql data base is commonly used in current cloud platform
Planting database schema, its built-in copy function is that structure is large-scale, the basis of high performance application.
The data of Mysql are distributed to multiple system up, and the mechanism of this distribution, is by by Mysql
The data of a certain main frame copy on other main frame (slaves), and re-execute one time come real
Existing.In reproduction process, a server serves as master server, and other server one or more
Serve as from server.Master server is by more newly written binary log (binlog) file, and safeguards
One index of file circulates with trace log.These daily records can record and be sent to from server
Update.When one from server connect master server time, it notice master server from server in day
The position being successfully updated for the last time read in will.From appointing that server reception occurs from that time
What updates, and then blocks and waits that master server notifies new renewal.
The client/server of Mysql data base achieves leader follower replication and the read and write abruption function of data base,
On the one hand ensure that the high availability of system, on the other hand improve the concurrent load capacity of data base.
The concordance ibid needing to ensure data realized and the transparent requirement to user.
The key point of the method for building up realizing Mysql data base's principal and subordinate's server is substantially: from server
Need the data of principal and subordinate's server data snapshot point, ensure that it is from these snapshot main clothes of point
Business device synchronizes.Currently employed is that software tool based on MySQL master server is created snapshot
The methods such as point, by data snapshot copy extremely from server, and then realize and the synchronization of main service.
Although this method for building up can ensure that the concordance of data, and ensures that database service is not interrupted.
But when master server data volume is the biggest, create and the data volume of master server time-consuming from server
Being directly proportional, the copy data that require a great deal of time are to from server;Further, since main service
Device is identical with each from server data, wastes disk space to a certain extent.
In view of this, it is necessary to cloud platform Mysql data base principal and subordinate's server in prior art
Method for building up is improved, to solve the problems referred to above.
Summary of the invention
It is an object of the invention to disclose the method for building up of a kind of cloud platform data base's principal and subordinate's server, use
To realize ensureing master server and the data consistency from server and not blocking master server
On the premise of service, reduce the time-consuming waste simultaneously reducing disk space created from server.
For achieving the above object, the invention provides the foundation of a kind of cloud platform data base's principal and subordinate's server
Method, described method for building up includes:
S1, select or create master server and its being customized is configured, and master server is made fast
According to operation;
S2, create from server data dish according to the snapshot in step S1, start from server;
S3, only record from the binlog literary composition corresponding after database service successfully starts up for the first time of server
Part name and positional information;
S4, read respectively according to the binlog filename in step S3 and positional information master server and from
The CRC32 check value of database manipulation record corresponding in the binlog file of server is also sentenced
Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main
Server and simultaneously operating from server, repeat described step S1 to step S3, until success
Set up master server and from server.
As a further improvement on the present invention, the customization in step S1 configuration include global variable configuration,
Cache flush configuration, distributed transaction configuration and the verification configuration of leader follower replication event.
As a further improvement on the present invention, in step S2 the data in master server data disks with support
The distributed memory system of snapshot creates.
As a further improvement on the present invention, distributed memory system includes that the storage of Ceph distributed block is
System.
As a further improvement on the present invention, step S4 also includes when according to the binlog literary composition in step S3
Part name and positional information read master server and data corresponding from the binlog file of server respectively
The CRC32 check value of storehouse operation note when carrying out judging and being different, deletes master server data disks also
It is not turned on the step of master server and the simultaneously operating from server.
As a further improvement on the present invention, cloud platform data base is MySQL database.
Compared with prior art, the invention has the beneficial effects as follows: in the present invention by establishment
The form of the data acquisition snapshot in master server data disks stores, and realizes number in disk aspect
According to rapid copy, solve taking of disk;Meanwhile, believe according to binlog filename and position
Breath reads master server and database manipulation corresponding from the binlog file of server note respectively
The CRC32 check value of record also judges, and only create master server and from clothes when judging consistent
Business device, quickly sets up principal and subordinate's server in the situation not blocking master server service, and ensure that master
The concordance of data from server.
Detailed description of the invention
Below in conjunction with each embodiment, the present invention is described in detail, but it should explanation, these
Embodiment not limitation of the present invention, those of ordinary skill in the art are according to these embodiment institutes
Equivalent transformation in work energy, method or structure or replacement, belong to the protection model of the present invention
Within enclosing.
In the present embodiment, the method for building up of a kind of cloud platform data base's principal and subordinate's server, described foundation
Method comprises the following steps:
Step S1, select or create master server and its being customized is configured, and to master server
Make snapshot operation.In this step S1 customization configuration include global variable configuration, cache flush configuration,
Distributed transaction configuration and the verification configuration of leader follower replication event.Concrete, the operation generation of this customization configuration
Code is as follows:
Sync_binlog=1
Innodb_flush_log_at_trx_commit=1
Innodb_support_xa=1
Binlog_checksum=crc32.
The Introduction on Principle that MySQL replicates is as follows: MySQL master library writes binlog when affairs are submitted to, and
Control binlog by sync_binlog parameter to flush to disk and " land ".And for storehouse by IO line
Journey pulls binlog from master library, and recorded in the relay log of this locality;Again will by local SQL thread
Market demand in relay log is in local data base.
The binlog that MySQL provides a sync_binlog parameter to control database brushes on disk
Go.Although binlog also has binlog cache, but MySQL not control binlog cache is same
Step is to the relevant consideration of file system cache.So be not related to binlog cache here.
Acquiescence, sync_binlog=0, represent that MySQL does not control the refreshing of binlog, by file system
Oneself control the refreshing of its caching.
If sync_binlog > 0, representing that every sync_binlog affairs are submitted to, MySQL calls file
Caching brush is gone down by the refresh operation of system.Safest is exactly sync_binlog=1, represents every time
Affairs are submitted to, and MySQL can go down binlog brush.Like this, at the main frame at data base place
Operating system is damaged or in the case of power down suddenly, system its is possible to lose the data of 1 affairs.
Although but binlog is order IO, but arranging sync_binlog=1, multiple affairs are submitted to simultaneously,
The biggest affects MySQL and IO performance.Although can be delayed by the patch of group commit
Solve, but the too high impact on IO of frequency refreshed is the biggest.
So the sync_binlog that a lot of MySQL DBA are arranged not is safest 1, but 100
Or 0.So sacrifice certain concordance, it is possible to obtain higher concurrent and performance.
In the present embodiment, this cloud platform data base is MySQL database.MySQL database
Supporting unidirectional, asynchronous replication, in reproduction process, a server is as master server, and one or
Person's other servers multiple serve as from server.Master server by more newly written binary log file,
And safeguard that an index of journal file circulates with trace log.It is connected to main clothes from server when one
During business device, daily record, read last from server notice master server from server and be successfully updated
Position.Receive any renewal occurred from that time from server, then block and wait main service
Device notice updates next time.
The principal and subordinate's server set up by configuration as above can ensure that master server or from service different
Often, such as the consistency problem of data in the principal and subordinate's server after unexpected power-down rebooting, it is distributed
Advantage when storage system or storage medium are positioned at long-range property server becomes apparent from.
Step S2, create from server data dish according to the snapshot in step S1, and to described from service
Start from server after the configuration of device being customized of data disks.In this step S2 master server data disks with
The distributed memory system supporting snapshot creates;Preferably, this distributed memory system includes Ceph
Block storage system.Ceph block storage system is the distributed memory system of a Linux PB rank, its
Framework probably can be divided into four parts: client (data user), meta data server (caching and
Synchronize distributed meta data), object storage cluster (data and metadata are stored as object),
Cluster monitor (execution function for monitoring).
Create and store the data in data base with snapshot form, it is possible to resolve taking of the disk in server
Space.Server herein refers to the server of physical aspect, and disk refers to the disk of physical aspect, disk battle array
Row (RAID) or the storage medium being made up of non-volatile memory device (NV-Memory).
Step S3, only record from server after database service successfully starts up for the first time corresponding
Binlog filename and positional information.
Step S4, read respectively according to the binlog filename in step S3 and positional information master server and
The CRC32 check value of database manipulation record corresponding from the binlog file of server is also sentenced
Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main
Server and simultaneously operating from server, repeat described step S1 to step S3, until successfully building
Found master server and from server.
Preferably, this step S4 also includes when dividing according to the binlog filename in step S3 and positional information
Du Qu master server and database manipulation record corresponding from the binlog file of server
When CRC32 check value carrying out judges and be different, delete master server data disks and be also not turned on master server
Step with the simultaneously operating from server.
CRC32 (Cyclic Redundancy Check) is a kind of CRC.CRC32 school
Test that to have error detecing capability extremely strong, the advantage that expense is little, it is easy to realize with encoder and testing circuit.From it
From the point of view of error detecing capability, it the probability of the most detectable mistake be only less than 0.0047%.From performance and
Consider in expense, be all far superior to the mode such as even-odd check and arithmetic sum verification.Use check (CRC) 32 school
The method testing value, it is ensured that master server and the concordance of data from server.
In the present embodiment, the master server being established with from server, all contain Agent program,
This Agent program can be responsible for MySQL state information searching in the configuration of binlog file synchronization and server
Deng work.
The a series of detailed description of those listed above is only for the feasibility embodiment of the present invention
Illustrate, they also are not used to limit the scope of the invention, all without departing from skill of the present invention
Equivalent implementations or change that spirit is made should be included within the scope of the present invention.
It is obvious to a person skilled in the art that the invention is not restricted to the thin of above-mentioned one exemplary embodiment
Joint, and without departing from the spirit or essential characteristics of the present invention, it is possible to other concrete shape
Formula realizes the present invention.Therefore, no matter from the point of view of which point, embodiment all should be regarded as exemplary,
And be nonrestrictive, the scope of the present invention is limited by claims rather than described above, because of
This is intended to include at this all changes fallen in the implication of equivalency and scope of claim
In bright.
Moreover, it will be appreciated that although this specification is been described by according to embodiment, but the most each
Embodiment only comprises an independent technical scheme, and this narrating mode of description is only is clear
Chu Qijian, those skilled in the art should be using description as an entirety, the technology in each embodiment
Scheme can also form, through appropriately combined, other embodiments that it will be appreciated by those skilled in the art that.
Claims (6)
1. the method for building up of cloud platform data base's principal and subordinate's server, it is characterised in that described foundation
Method includes:
S1, select or create master server and its being customized is configured, and master server is made fast
According to operation;
S2, create from server data dish according to the snapshot in step S1, start from server;
S3, only record from the binlog literary composition corresponding after database service successfully starts up for the first time of server
Part name and positional information;
S4, read respectively according to the binlog filename in step S3 and positional information master server and from
The CRC32 check value of database manipulation record corresponding in the binlog file of server is also sentenced
Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main
Server and simultaneously operating from server, repeat described step S1 to step S3, until success
Set up master server and from server.
Method for building up the most according to claim 1, it is characterised in that determining in described step S1
Inhibition and generation configuration includes global variable configuration, cache flush configuration, distributed transaction configuration and leader follower replication thing
Part verification configuration.
Method for building up the most according to claim 1, it is characterised in that main clothes in described step S2
Business device data disks is to support that the distributed memory system of snapshot creates.
Method for building up the most according to claim 3, it is characterised in that described distributed storage system bag
Include the distributed block storage system of Ceph.
Method for building up the most according to claim 1, it is characterised in that described step S4 also includes
When reading master server respectively and from server according to the binlog filename in step S3 and positional information
Binlog file in the CRC32 check value carrying out of corresponding database manipulation record judge and not
Meanwhile, delete master server data disks and be not turned on the step of master server and the simultaneously operating from server
Suddenly.
Method for building up the most according to claim 1, it is characterised in that described cloud platform data base is
MySQL database.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2015110346852 | 2015-12-31 | ||
CN201511034685 | 2015-12-31 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105955989A true CN105955989A (en) | 2016-09-21 |
CN105955989B CN105955989B (en) | 2020-12-22 |
Family
ID=56917642
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610244979.6A Active CN105955989B (en) | 2015-12-31 | 2016-04-19 | Method for establishing master server and slave server of cloud platform database |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105955989B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108763578A (en) * | 2018-06-07 | 2018-11-06 | 腾讯科技(深圳)有限公司 | A kind of newer method of index file and server |
CN108804463A (en) * | 2017-05-03 | 2018-11-13 | 杭州海康威视数字技术股份有限公司 | A kind of method of data synchronization of MySQL database, device and electronic equipment |
CN112445780A (en) * | 2020-11-27 | 2021-03-05 | 上海上讯信息技术股份有限公司 | Master-slave architecture deployment method and device based on snapshot |
CN114020850A (en) * | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
US11550953B2 (en) | 2020-09-16 | 2023-01-10 | Saudi Arabian Oil Company | Preserving cloud anonymity |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6247141B1 (en) * | 1998-09-24 | 2001-06-12 | Telefonaktiebolaget Lm Ericsson (Publ) | Protocol for providing replicated servers in a client-server system |
CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
CN103530290A (en) * | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
CN103905220A (en) * | 2012-12-25 | 2014-07-02 | 腾讯科技(北京)有限公司 | Data synchronization processing method and system |
CN104252500A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out fault repairing on database management platform |
CN104252502A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out data migration on database management platform |
CN104506625A (en) * | 2014-12-22 | 2015-04-08 | 国云科技股份有限公司 | Method for improving reliability of metadata nodes of cloud databases |
CN104657382A (en) * | 2013-11-21 | 2015-05-27 | 阿里巴巴集团控股有限公司 | Method and device for detecting consistency of data of MySQL master and slave servers |
CN104679614A (en) * | 2015-03-31 | 2015-06-03 | 成都文武信息技术有限公司 | Database disaster backup system |
US9152686B2 (en) * | 2012-12-21 | 2015-10-06 | Zetta Inc. | Asynchronous replication correctness validation |
-
2016
- 2016-04-19 CN CN201610244979.6A patent/CN105955989B/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6247141B1 (en) * | 1998-09-24 | 2001-06-12 | Telefonaktiebolaget Lm Ericsson (Publ) | Protocol for providing replicated servers in a client-server system |
CN103530290A (en) * | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
US9152686B2 (en) * | 2012-12-21 | 2015-10-06 | Zetta Inc. | Asynchronous replication correctness validation |
CN103905220A (en) * | 2012-12-25 | 2014-07-02 | 腾讯科技(北京)有限公司 | Data synchronization processing method and system |
CN104252500A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out fault repairing on database management platform |
CN104252502A (en) * | 2013-06-29 | 2014-12-31 | 北京新媒传信科技有限公司 | Method and device for carrying out data migration on database management platform |
CN104657382A (en) * | 2013-11-21 | 2015-05-27 | 阿里巴巴集团控股有限公司 | Method and device for detecting consistency of data of MySQL master and slave servers |
CN104506625A (en) * | 2014-12-22 | 2015-04-08 | 国云科技股份有限公司 | Method for improving reliability of metadata nodes of cloud databases |
CN104679614A (en) * | 2015-03-31 | 2015-06-03 | 成都文武信息技术有限公司 | Database disaster backup system |
Non-Patent Citations (3)
Title |
---|
LINWATERBIN: "腾讯游戏数据自愈服务方案", 《CSDN博客HTTPS://BLOG.CSDN.NET/DBA_WATERBIN/ARTICLE/DETAILS/43608587》 * |
徐阳等: "两种方法实现DNS主从服务器数据同步", 《中国教育网络》 * |
陈东江: "基于数据同步的集群系统不间断服务的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108804463A (en) * | 2017-05-03 | 2018-11-13 | 杭州海康威视数字技术股份有限公司 | A kind of method of data synchronization of MySQL database, device and electronic equipment |
CN108763578A (en) * | 2018-06-07 | 2018-11-06 | 腾讯科技(深圳)有限公司 | A kind of newer method of index file and server |
US11550953B2 (en) | 2020-09-16 | 2023-01-10 | Saudi Arabian Oil Company | Preserving cloud anonymity |
CN112445780A (en) * | 2020-11-27 | 2021-03-05 | 上海上讯信息技术股份有限公司 | Master-slave architecture deployment method and device based on snapshot |
WO2022110425A1 (en) * | 2020-11-27 | 2022-06-02 | 上海上讯信息技术股份有限公司 | Snapshot-based master-slave architecture deployment method and device |
US11966296B2 (en) | 2020-11-27 | 2024-04-23 | Shanghai Suninfo Information Technology Co., Ltd. | Master-slave architecture deployment method and device based on snapshot |
CN114020850A (en) * | 2022-01-05 | 2022-02-08 | 深圳市明源云科技有限公司 | Database data synchronization method, device, equipment and readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN105955989B (en) | 2020-12-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10503616B2 (en) | Periodic data replication | |
CN103842969B (en) | Information processing system | |
Baker et al. | Megastore: Providing scalable, highly available storage for interactive services. | |
US10248356B2 (en) | Using scratch extents to facilitate copying operations in an append-only storage system | |
US7330859B2 (en) | Database backup system using data and user-defined routines replicators for maintaining a copy of database on a secondary server | |
US8996458B2 (en) | High volume, high speed adaptive data replication | |
CN105955989A (en) | Method for establishing master and slave servers of cloud platform database | |
Aiyer et al. | Storage infrastructure behind Facebook messages: Using HBase at scale. | |
EP2687987A1 (en) | Method, system and serving node for data backup and recovery | |
US20130262389A1 (en) | Parallel Backup for Distributed Database System Environments | |
US11093387B1 (en) | Garbage collection based on transmission object models | |
CN107209704A (en) | Detect the write-in lost | |
CN107835983A (en) | Backup-and-restore is carried out in distributed data base using consistent database snapshot | |
US10649980B2 (en) | Methods and systems for resilient, durable, scalable, and consistent distributed timeline data store | |
US20150254320A1 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
CN102591982A (en) | Method and system of performing incremental sql server database backups | |
EP2380090B1 (en) | Data integrity in a database environment through background synchronization | |
JP2015504218A (en) | Distributed database with modular blocks and associated log files | |
CN106062742A (en) | Systems and methods for improving snapshot performance | |
CN105843702A (en) | Method and device for data backup | |
CN101986276B (en) | Methods and systems for storing and recovering files and server | |
CN101901275A (en) | Distributed storage system and method thereof | |
US9619322B2 (en) | Erasure-coding extents in an append-only storage system | |
CN102682110A (en) | High-performance cache design method orienting to massive spatial information | |
CN100504800C (en) | Method for snapshot of magnetic disc |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB02 | Change of applicant information |
Address after: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu Applicant after: Huayun data holding group Co., Ltd Address before: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu Applicant before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd. |
|
CB02 | Change of applicant information | ||
GR01 | Patent grant | ||
GR01 | Patent grant |