CN105955989A - Method for establishing master and slave servers of cloud platform database - Google Patents

Method for establishing master and slave servers of cloud platform database Download PDF

Info

Publication number
CN105955989A
CN105955989A CN201610244979.6A CN201610244979A CN105955989A CN 105955989 A CN105955989 A CN 105955989A CN 201610244979 A CN201610244979 A CN 201610244979A CN 105955989 A CN105955989 A CN 105955989A
Authority
CN
China
Prior art keywords
server
master
binlog
master server
building
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610244979.6A
Other languages
Chinese (zh)
Other versions
CN105955989B (en
Inventor
许广彬
郑军
张银滨
刘晓欣
刘志坤
张欢
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuxi Huayun Data Technology Service Co Ltd
Original Assignee
Wuxi Huayun Data Technology Service Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuxi Huayun Data Technology Service Co Ltd filed Critical Wuxi Huayun Data Technology Service Co Ltd
Publication of CN105955989A publication Critical patent/CN105955989A/en
Application granted granted Critical
Publication of CN105955989B publication Critical patent/CN105955989B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)

Abstract

The invention provides a method for establishing master and slave servers of a cloud platform database. The method comprises the steps that at the step S1, master and slave servers are selected or established, customized configuration is carried out to the servers, and a snapshot operation is implemented; at the step S2, a snapshot at the step S1 is used to establish a data disk of the slave server, and the slave server is started; at the step S3, a name and position information of a corresponding binlog document existing after a database service of the slave server is started successfully for the first time are recorded; and at the step S4, CRC32 verification values of database operation records corresponding to the binlog documents of the master and slave servers are read respectively according to the name and the position information of the binlog document obtained at the step S3, and the verification values are judged, wherein an synchronous operation of the master and slave servers is started if the verification values are same, and the synchronous operation of the master and slave servers are not started, and the steps from S1 to S3 are executed repeatedly till the master and slave servers are successfully established if the verification values are different. The method provided by the invention has the advantages that disk occupied space can be effectively reduced; the master and slave servers can be established rapidly without blocking in services of the master server; and consistency in data of the master and slave servers is ensured.

Description

A kind of method for building up of cloud platform data base's principal and subordinate's server
Technical field
The present invention relates to field of cloud computer technology, particularly relate to a kind of cloud platform data base's principal and subordinate's server Method for building up.
Background technology
In cloud platform, data base realizes virtual computing, virtual memory, development environment etc. for platform Multiple offer data of quoting store and call.Mysql data base is commonly used in current cloud platform Planting database schema, its built-in copy function is that structure is large-scale, the basis of high performance application. The data of Mysql are distributed to multiple system up, and the mechanism of this distribution, is by by Mysql The data of a certain main frame copy on other main frame (slaves), and re-execute one time come real Existing.In reproduction process, a server serves as master server, and other server one or more Serve as from server.Master server is by more newly written binary log (binlog) file, and safeguards One index of file circulates with trace log.These daily records can record and be sent to from server Update.When one from server connect master server time, it notice master server from server in day The position being successfully updated for the last time read in will.From appointing that server reception occurs from that time What updates, and then blocks and waits that master server notifies new renewal.
The client/server of Mysql data base achieves leader follower replication and the read and write abruption function of data base, On the one hand ensure that the high availability of system, on the other hand improve the concurrent load capacity of data base. The concordance ibid needing to ensure data realized and the transparent requirement to user.
The key point of the method for building up realizing Mysql data base's principal and subordinate's server is substantially: from server Need the data of principal and subordinate's server data snapshot point, ensure that it is from these snapshot main clothes of point Business device synchronizes.Currently employed is that software tool based on MySQL master server is created snapshot The methods such as point, by data snapshot copy extremely from server, and then realize and the synchronization of main service. Although this method for building up can ensure that the concordance of data, and ensures that database service is not interrupted. But when master server data volume is the biggest, create and the data volume of master server time-consuming from server Being directly proportional, the copy data that require a great deal of time are to from server;Further, since main service Device is identical with each from server data, wastes disk space to a certain extent.
In view of this, it is necessary to cloud platform Mysql data base principal and subordinate's server in prior art Method for building up is improved, to solve the problems referred to above.
Summary of the invention
It is an object of the invention to disclose the method for building up of a kind of cloud platform data base's principal and subordinate's server, use To realize ensureing master server and the data consistency from server and not blocking master server On the premise of service, reduce the time-consuming waste simultaneously reducing disk space created from server.
For achieving the above object, the invention provides the foundation of a kind of cloud platform data base's principal and subordinate's server Method, described method for building up includes:
S1, select or create master server and its being customized is configured, and master server is made fast According to operation;
S2, create from server data dish according to the snapshot in step S1, start from server;
S3, only record from the binlog literary composition corresponding after database service successfully starts up for the first time of server Part name and positional information;
S4, read respectively according to the binlog filename in step S3 and positional information master server and from The CRC32 check value of database manipulation record corresponding in the binlog file of server is also sentenced Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main Server and simultaneously operating from server, repeat described step S1 to step S3, until success Set up master server and from server.
As a further improvement on the present invention, the customization in step S1 configuration include global variable configuration, Cache flush configuration, distributed transaction configuration and the verification configuration of leader follower replication event.
As a further improvement on the present invention, in step S2 the data in master server data disks with support The distributed memory system of snapshot creates.
As a further improvement on the present invention, distributed memory system includes that the storage of Ceph distributed block is System.
As a further improvement on the present invention, step S4 also includes when according to the binlog literary composition in step S3 Part name and positional information read master server and data corresponding from the binlog file of server respectively The CRC32 check value of storehouse operation note when carrying out judging and being different, deletes master server data disks also It is not turned on the step of master server and the simultaneously operating from server.
As a further improvement on the present invention, cloud platform data base is MySQL database.
Compared with prior art, the invention has the beneficial effects as follows: in the present invention by establishment The form of the data acquisition snapshot in master server data disks stores, and realizes number in disk aspect According to rapid copy, solve taking of disk;Meanwhile, believe according to binlog filename and position Breath reads master server and database manipulation corresponding from the binlog file of server note respectively The CRC32 check value of record also judges, and only create master server and from clothes when judging consistent Business device, quickly sets up principal and subordinate's server in the situation not blocking master server service, and ensure that master The concordance of data from server.
Detailed description of the invention
Below in conjunction with each embodiment, the present invention is described in detail, but it should explanation, these Embodiment not limitation of the present invention, those of ordinary skill in the art are according to these embodiment institutes Equivalent transformation in work energy, method or structure or replacement, belong to the protection model of the present invention Within enclosing.
In the present embodiment, the method for building up of a kind of cloud platform data base's principal and subordinate's server, described foundation Method comprises the following steps:
Step S1, select or create master server and its being customized is configured, and to master server Make snapshot operation.In this step S1 customization configuration include global variable configuration, cache flush configuration, Distributed transaction configuration and the verification configuration of leader follower replication event.Concrete, the operation generation of this customization configuration Code is as follows:
Sync_binlog=1
Innodb_flush_log_at_trx_commit=1
Innodb_support_xa=1
Binlog_checksum=crc32.
The Introduction on Principle that MySQL replicates is as follows: MySQL master library writes binlog when affairs are submitted to, and Control binlog by sync_binlog parameter to flush to disk and " land ".And for storehouse by IO line Journey pulls binlog from master library, and recorded in the relay log of this locality;Again will by local SQL thread Market demand in relay log is in local data base.
The binlog that MySQL provides a sync_binlog parameter to control database brushes on disk Go.Although binlog also has binlog cache, but MySQL not control binlog cache is same Step is to the relevant consideration of file system cache.So be not related to binlog cache here.
Acquiescence, sync_binlog=0, represent that MySQL does not control the refreshing of binlog, by file system Oneself control the refreshing of its caching.
If sync_binlog > 0, representing that every sync_binlog affairs are submitted to, MySQL calls file Caching brush is gone down by the refresh operation of system.Safest is exactly sync_binlog=1, represents every time Affairs are submitted to, and MySQL can go down binlog brush.Like this, at the main frame at data base place Operating system is damaged or in the case of power down suddenly, system its is possible to lose the data of 1 affairs. Although but binlog is order IO, but arranging sync_binlog=1, multiple affairs are submitted to simultaneously, The biggest affects MySQL and IO performance.Although can be delayed by the patch of group commit Solve, but the too high impact on IO of frequency refreshed is the biggest.
So the sync_binlog that a lot of MySQL DBA are arranged not is safest 1, but 100 Or 0.So sacrifice certain concordance, it is possible to obtain higher concurrent and performance.
In the present embodiment, this cloud platform data base is MySQL database.MySQL database Supporting unidirectional, asynchronous replication, in reproduction process, a server is as master server, and one or Person's other servers multiple serve as from server.Master server by more newly written binary log file, And safeguard that an index of journal file circulates with trace log.It is connected to main clothes from server when one During business device, daily record, read last from server notice master server from server and be successfully updated Position.Receive any renewal occurred from that time from server, then block and wait main service Device notice updates next time.
The principal and subordinate's server set up by configuration as above can ensure that master server or from service different Often, such as the consistency problem of data in the principal and subordinate's server after unexpected power-down rebooting, it is distributed Advantage when storage system or storage medium are positioned at long-range property server becomes apparent from.
Step S2, create from server data dish according to the snapshot in step S1, and to described from service Start from server after the configuration of device being customized of data disks.In this step S2 master server data disks with The distributed memory system supporting snapshot creates;Preferably, this distributed memory system includes Ceph Block storage system.Ceph block storage system is the distributed memory system of a Linux PB rank, its Framework probably can be divided into four parts: client (data user), meta data server (caching and Synchronize distributed meta data), object storage cluster (data and metadata are stored as object), Cluster monitor (execution function for monitoring).
Create and store the data in data base with snapshot form, it is possible to resolve taking of the disk in server Space.Server herein refers to the server of physical aspect, and disk refers to the disk of physical aspect, disk battle array Row (RAID) or the storage medium being made up of non-volatile memory device (NV-Memory).
Step S3, only record from server after database service successfully starts up for the first time corresponding Binlog filename and positional information.
Step S4, read respectively according to the binlog filename in step S3 and positional information master server and The CRC32 check value of database manipulation record corresponding from the binlog file of server is also sentenced Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main Server and simultaneously operating from server, repeat described step S1 to step S3, until successfully building Found master server and from server.
Preferably, this step S4 also includes when dividing according to the binlog filename in step S3 and positional information Du Qu master server and database manipulation record corresponding from the binlog file of server When CRC32 check value carrying out judges and be different, delete master server data disks and be also not turned on master server Step with the simultaneously operating from server.
CRC32 (Cyclic Redundancy Check) is a kind of CRC.CRC32 school Test that to have error detecing capability extremely strong, the advantage that expense is little, it is easy to realize with encoder and testing circuit.From it From the point of view of error detecing capability, it the probability of the most detectable mistake be only less than 0.0047%.From performance and Consider in expense, be all far superior to the mode such as even-odd check and arithmetic sum verification.Use check (CRC) 32 school The method testing value, it is ensured that master server and the concordance of data from server.
In the present embodiment, the master server being established with from server, all contain Agent program, This Agent program can be responsible for MySQL state information searching in the configuration of binlog file synchronization and server Deng work.
The a series of detailed description of those listed above is only for the feasibility embodiment of the present invention Illustrate, they also are not used to limit the scope of the invention, all without departing from skill of the present invention Equivalent implementations or change that spirit is made should be included within the scope of the present invention.
It is obvious to a person skilled in the art that the invention is not restricted to the thin of above-mentioned one exemplary embodiment Joint, and without departing from the spirit or essential characteristics of the present invention, it is possible to other concrete shape Formula realizes the present invention.Therefore, no matter from the point of view of which point, embodiment all should be regarded as exemplary, And be nonrestrictive, the scope of the present invention is limited by claims rather than described above, because of This is intended to include at this all changes fallen in the implication of equivalency and scope of claim In bright.
Moreover, it will be appreciated that although this specification is been described by according to embodiment, but the most each Embodiment only comprises an independent technical scheme, and this narrating mode of description is only is clear Chu Qijian, those skilled in the art should be using description as an entirety, the technology in each embodiment Scheme can also form, through appropriately combined, other embodiments that it will be appreciated by those skilled in the art that.

Claims (6)

1. the method for building up of cloud platform data base's principal and subordinate's server, it is characterised in that described foundation Method includes:
S1, select or create master server and its being customized is configured, and master server is made fast According to operation;
S2, create from server data dish according to the snapshot in step S1, start from server;
S3, only record from the binlog literary composition corresponding after database service successfully starts up for the first time of server Part name and positional information;
S4, read respectively according to the binlog filename in step S3 and positional information master server and from The CRC32 check value of database manipulation record corresponding in the binlog file of server is also sentenced Disconnected, if identical, then open master server and the simultaneously operating from server;If it is different, be then not turned on main Server and simultaneously operating from server, repeat described step S1 to step S3, until success Set up master server and from server.
Method for building up the most according to claim 1, it is characterised in that determining in described step S1 Inhibition and generation configuration includes global variable configuration, cache flush configuration, distributed transaction configuration and leader follower replication thing Part verification configuration.
Method for building up the most according to claim 1, it is characterised in that main clothes in described step S2 Business device data disks is to support that the distributed memory system of snapshot creates.
Method for building up the most according to claim 3, it is characterised in that described distributed storage system bag Include the distributed block storage system of Ceph.
Method for building up the most according to claim 1, it is characterised in that described step S4 also includes When reading master server respectively and from server according to the binlog filename in step S3 and positional information Binlog file in the CRC32 check value carrying out of corresponding database manipulation record judge and not Meanwhile, delete master server data disks and be not turned on the step of master server and the simultaneously operating from server Suddenly.
Method for building up the most according to claim 1, it is characterised in that described cloud platform data base is MySQL database.
CN201610244979.6A 2015-12-31 2016-04-19 Method for establishing master server and slave server of cloud platform database Active CN105955989B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2015110346852 2015-12-31
CN201511034685 2015-12-31

Publications (2)

Publication Number Publication Date
CN105955989A true CN105955989A (en) 2016-09-21
CN105955989B CN105955989B (en) 2020-12-22

Family

ID=56917642

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610244979.6A Active CN105955989B (en) 2015-12-31 2016-04-19 Method for establishing master server and slave server of cloud platform database

Country Status (1)

Country Link
CN (1) CN105955989B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108763578A (en) * 2018-06-07 2018-11-06 腾讯科技(深圳)有限公司 A kind of newer method of index file and server
CN108804463A (en) * 2017-05-03 2018-11-13 杭州海康威视数字技术股份有限公司 A kind of method of data synchronization of MySQL database, device and electronic equipment
CN112445780A (en) * 2020-11-27 2021-03-05 上海上讯信息技术股份有限公司 Master-slave architecture deployment method and device based on snapshot
CN114020850A (en) * 2022-01-05 2022-02-08 深圳市明源云科技有限公司 Database data synchronization method, device, equipment and readable storage medium
US11550953B2 (en) 2020-09-16 2023-01-10 Saudi Arabian Oil Company Preserving cloud anonymity

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6247141B1 (en) * 1998-09-24 2001-06-12 Telefonaktiebolaget Lm Ericsson (Publ) Protocol for providing replicated servers in a client-server system
CN103020243A (en) * 2012-12-18 2013-04-03 新浪网技术(中国)有限公司 Method and device for master-slave replication structure replacement of database
CN103530290A (en) * 2012-07-03 2014-01-22 深圳市腾讯计算机系统有限公司 Method and system for data migration among databases
CN103905220A (en) * 2012-12-25 2014-07-02 腾讯科技(北京)有限公司 Data synchronization processing method and system
CN104252500A (en) * 2013-06-29 2014-12-31 北京新媒传信科技有限公司 Method and device for carrying out fault repairing on database management platform
CN104252502A (en) * 2013-06-29 2014-12-31 北京新媒传信科技有限公司 Method and device for carrying out data migration on database management platform
CN104506625A (en) * 2014-12-22 2015-04-08 国云科技股份有限公司 Method for improving reliability of metadata nodes of cloud databases
CN104657382A (en) * 2013-11-21 2015-05-27 阿里巴巴集团控股有限公司 Method and device for detecting consistency of data of MySQL master and slave servers
CN104679614A (en) * 2015-03-31 2015-06-03 成都文武信息技术有限公司 Database disaster backup system
US9152686B2 (en) * 2012-12-21 2015-10-06 Zetta Inc. Asynchronous replication correctness validation

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6247141B1 (en) * 1998-09-24 2001-06-12 Telefonaktiebolaget Lm Ericsson (Publ) Protocol for providing replicated servers in a client-server system
CN103530290A (en) * 2012-07-03 2014-01-22 深圳市腾讯计算机系统有限公司 Method and system for data migration among databases
CN103020243A (en) * 2012-12-18 2013-04-03 新浪网技术(中国)有限公司 Method and device for master-slave replication structure replacement of database
US9152686B2 (en) * 2012-12-21 2015-10-06 Zetta Inc. Asynchronous replication correctness validation
CN103905220A (en) * 2012-12-25 2014-07-02 腾讯科技(北京)有限公司 Data synchronization processing method and system
CN104252500A (en) * 2013-06-29 2014-12-31 北京新媒传信科技有限公司 Method and device for carrying out fault repairing on database management platform
CN104252502A (en) * 2013-06-29 2014-12-31 北京新媒传信科技有限公司 Method and device for carrying out data migration on database management platform
CN104657382A (en) * 2013-11-21 2015-05-27 阿里巴巴集团控股有限公司 Method and device for detecting consistency of data of MySQL master and slave servers
CN104506625A (en) * 2014-12-22 2015-04-08 国云科技股份有限公司 Method for improving reliability of metadata nodes of cloud databases
CN104679614A (en) * 2015-03-31 2015-06-03 成都文武信息技术有限公司 Database disaster backup system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
LINWATERBIN: "腾讯游戏数据自愈服务方案", 《CSDN博客HTTPS://BLOG.CSDN.NET/DBA_WATERBIN/ARTICLE/DETAILS/43608587》 *
徐阳等: "两种方法实现DNS主从服务器数据同步", 《中国教育网络》 *
陈东江: "基于数据同步的集群系统不间断服务的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108804463A (en) * 2017-05-03 2018-11-13 杭州海康威视数字技术股份有限公司 A kind of method of data synchronization of MySQL database, device and electronic equipment
CN108763578A (en) * 2018-06-07 2018-11-06 腾讯科技(深圳)有限公司 A kind of newer method of index file and server
US11550953B2 (en) 2020-09-16 2023-01-10 Saudi Arabian Oil Company Preserving cloud anonymity
CN112445780A (en) * 2020-11-27 2021-03-05 上海上讯信息技术股份有限公司 Master-slave architecture deployment method and device based on snapshot
WO2022110425A1 (en) * 2020-11-27 2022-06-02 上海上讯信息技术股份有限公司 Snapshot-based master-slave architecture deployment method and device
US11966296B2 (en) 2020-11-27 2024-04-23 Shanghai Suninfo Information Technology Co., Ltd. Master-slave architecture deployment method and device based on snapshot
CN114020850A (en) * 2022-01-05 2022-02-08 深圳市明源云科技有限公司 Database data synchronization method, device, equipment and readable storage medium

Also Published As

Publication number Publication date
CN105955989B (en) 2020-12-22

Similar Documents

Publication Publication Date Title
US10503616B2 (en) Periodic data replication
CN103842969B (en) Information processing system
Baker et al. Megastore: Providing scalable, highly available storage for interactive services.
US10248356B2 (en) Using scratch extents to facilitate copying operations in an append-only storage system
US7330859B2 (en) Database backup system using data and user-defined routines replicators for maintaining a copy of database on a secondary server
US8996458B2 (en) High volume, high speed adaptive data replication
CN105955989A (en) Method for establishing master and slave servers of cloud platform database
Aiyer et al. Storage infrastructure behind Facebook messages: Using HBase at scale.
EP2687987A1 (en) Method, system and serving node for data backup and recovery
US20130262389A1 (en) Parallel Backup for Distributed Database System Environments
US11093387B1 (en) Garbage collection based on transmission object models
CN107209704A (en) Detect the write-in lost
CN107835983A (en) Backup-and-restore is carried out in distributed data base using consistent database snapshot
US10649980B2 (en) Methods and systems for resilient, durable, scalable, and consistent distributed timeline data store
US20150254320A1 (en) Using colocation hints to facilitate accessing a distributed data storage system
CN102591982A (en) Method and system of performing incremental sql server database backups
EP2380090B1 (en) Data integrity in a database environment through background synchronization
JP2015504218A (en) Distributed database with modular blocks and associated log files
CN106062742A (en) Systems and methods for improving snapshot performance
CN105843702A (en) Method and device for data backup
CN101986276B (en) Methods and systems for storing and recovering files and server
CN101901275A (en) Distributed storage system and method thereof
US9619322B2 (en) Erasure-coding extents in an append-only storage system
CN102682110A (en) High-performance cache design method orienting to massive spatial information
CN100504800C (en) Method for snapshot of magnetic disc

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu

Applicant after: Huayun data holding group Co., Ltd

Address before: 214000, Huayun data building, 6 B District, science and technology software park, Binhu West Road, Binhu District, Wuxi, Jiangsu

Applicant before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd.

CB02 Change of applicant information
GR01 Patent grant
GR01 Patent grant