CN106570007A - Method and equipment for data synchronization of distributed caching system - Google Patents

Method and equipment for data synchronization of distributed caching system Download PDF

Info

Publication number
CN106570007A
CN106570007A CN201510648329.3A CN201510648329A CN106570007A CN 106570007 A CN106570007 A CN 106570007A CN 201510648329 A CN201510648329 A CN 201510648329A CN 106570007 A CN106570007 A CN 106570007A
Authority
CN
China
Prior art keywords
data
data service
service equipment
equipment
main
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510648329.3A
Other languages
Chinese (zh)
Inventor
郑晓茵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201510648329.3A priority Critical patent/CN106570007A/en
Publication of CN106570007A publication Critical patent/CN106570007A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)

Abstract

The invention aims to provide a method and equipment for synchronization of a distributed caching system. The method comprises the steps of acquiring a distribution list from main cluster configuration service equipment, wherein the distribution list comprises information of the main cluster data service equipment which requires maintenance synchronization; based on the distribution list, acquiring a data log from the main cluster data service equipment, and generating synchronization task information based on the data log; and transmitting the synchronization task information to backup cluster data service equipment for synchronization, thereby continuously synchronizing the particular state of the data service equipment before shutdown to taking-over data service equipment when shutdown of the main cluster data service equipment or shutdown of the integral main cluster occurs, and preventing dirty data or data loss accordingly.

Description

For the method and apparatus of distributed cache system data syn-chronization
Technical field
The application is related to computer realm, more particularly to a kind of for distributed cache system data syn-chronization Technology.
Background technology
As local computer system is to the extension of distributed system, caching technology is led in Distributed Calculation It is widely used in domain, referred to as distributed caching.Distributed caching being capable of the reading of high-performance ground Data, can dynamically extend cache node, can find automatically and switch failure node, can from Dynamic equalization data subregion, and patterned administration interface, deployment and maintenance can be provided for user It is quite convenient to.Distributed caching has obtained widely should in field of distributed type, field of cloud calculation With.
At present, distributed cache system meets the appearance of types of applications by colony deployment way Calamity demand, including single part of two-shipper room list cluster, two-shipper room separate cluster, two-shipper room list cluster double, Two-shipper house-owner backs up cluster.For example, in the deployment way that two-shipper house-owner backs up cluster, there is one Main cluster and a backup cluster, respectively in two machine rooms, the deployment way of master backup cluster each other There is hidden danger in synchrodata, loss of data and dirty data can be caused.Popular says, when an affairs Data are being accessed, and data are being modified, and this modification is not also submitted to data base In, at this moment, another affairs also accesses this data, has then used this data.Because this Individual data are the data also do not submitted to, then this data that another affairs is read are dirty numbers According to.
The content of the invention
The purpose of the application is to provide a kind of method for distributed cache system data syn-chronization and sets It is standby, in caching in a distributed manner when machine occurs delaying in the main company-data service equipment for providing service, Backup set group energy proceeds complete data syn-chronization and does not lose data.
For this purpose, the one kind provided according to the one side of the application realizes that distributed cache system data are same The method of step, wherein, methods described includes:
Distribution list is obtained from main cluster configuration service equipment, the distribution list includes needing to safeguard same The information of the main company-data service equipment of step;
Based on the distribution list, data logging is obtained from the main company-data service equipment, and Synchronous task information is generated based on the data logging;
The synchronous task information is sent to backup cluster data service unit to synchronize.
Further, obtaining distribution list from main cluster configuration service equipment includes:
Connection with the main cluster configuration equipment is maintained by heartbeat mechanism;
Heart beating based on the main cluster configuration equipment is fed back, and obtains the distribution list.
Further, based on the distribution list, from the main company-data service equipment number is obtained Include according to daily record:
Based on the distribution list, the link information of the main company-data service equipment is obtained;
Based on the link information, set up with the main company-data service equipment and be connected, and obtain number According to daily record.
Further, obtaining the link information of the main company-data service equipment includes:
Based on the distribution list, from the main cluster configuration service equipment the main company-data is obtained The link information of service equipment, or main company-data service is obtained from the data of existing persistent storage The link information of equipment.
Further, the link information of the main company-data service equipment includes at least following arbitrary :
The net association of the main company-data service equipment;
The interface of the main company-data service equipment.
Further, methods described also includes:
From the main cluster configuration service equipment obtain the main company-data service equipment with confidence Breath;
Based on the configuration information, newly-built or renewal has the data of persistent storage;
Wherein, the configuration information of the main company-data service equipment includes:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment;
The data file information that the main company-data service equipment is stored;
The database engine serial number of the main company-data service equipment;
The database engine serial number of the alternate device of the main company-data service equipment.
Further, the synchronous task information is sent to backup cluster data service unit to carry out Synchronously include:
The configuration of the backup cluster data service unit is obtained from backup cluster configuration service equipment Information;
Based on the configuration information of the backup cluster data service unit, the synchronous task information is sent out The corresponding data service equipment of backup cluster is delivered to synchronize.
Further, the configuration information of the backup cluster data service unit includes at least following arbitrary :
The net association of the backup cluster data service unit;
The interface of the backup cluster data service unit;
The data file information that the backup cluster data service unit is stored;
The database engine serial number of the backup cluster data service unit;
The database engine serial number of the alternate device of the backup cluster data service unit.
Further, methods described also includes:
The information of newly-built backup cluster data storage device is obtained, and based on the information, from the master Company-data service equipment obtains snapshot data;
The snapshot data is sent into same to carry out to the data service unit of newly-built backup cluster Step.
Further, methods described also includes:
Data synchronization updating with other synchronizers is maintained by fixed time broadcast data mechanism.
Further, the data logging includes data operation information, data key and data value.
According to the application, on the other hand a kind of of offer realizes the same of distributed cache system data syn-chronization Step equipment, wherein, the synchronizer includes:
Distribution list acquisition device, it is described for obtaining distribution list from main cluster configuration service equipment Distribution list includes the information of the main company-data service equipment for needing to safeguard synchronous;
Data logging acquisition device, for based on the distribution list, from the main company-data service Data logging is obtained in equipment, and synchronous task information is generated based on the data logging;
Synchronizer, for by the synchronous task information send to backup cluster data service unit with Synchronize.
Further, the distribution list acquisition device is used for:
Connection with the main cluster configuration equipment is maintained by heartbeat mechanism;
Heart beating based on the main cluster configuration equipment is fed back, and obtains the distribution list.
Further, the data logging acquisition device is used for:
Based on the distribution list, the link information of the main company-data service equipment is obtained;
Based on the link information, set up with the main company-data service equipment and be connected, and obtain number According to daily record.
Further, the data logging acquisition device is used for:
Based on the distribution list, from the main cluster configuration service equipment the main company-data is obtained The link information of service equipment, or main company-data service is obtained from the data of existing persistent storage The link information of equipment.
Further, the link information of the main company-data service equipment includes at least following arbitrary :
The net association of the main company-data service equipment;
The interface of the main company-data service equipment.
Further, the synchronizer also includes:
Configuration information acquisition device, for obtaining the main cluster from the main cluster configuration service equipment The configuration information of data service unit, and based on the configuration information, newly-built or renewal has persistently Change the data of storage;
Wherein, the configuration information of the main company-data service equipment includes:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment;
The data file information that the main company-data service equipment is stored;
The database engine serial number of the main company-data service equipment;
The database engine serial number of the alternate device of the main company-data service equipment.
Further, the synchronizer is used for:
The configuration of the backup cluster data service unit is obtained from backup cluster configuration service equipment Information;
Based on the configuration information of the backup cluster data service unit, the synchronous task information is sent out The corresponding data service equipment of backup cluster is delivered to synchronize.
Further, the configuration information of the backup cluster data service unit includes at least following arbitrary :
The net association of the backup cluster data service unit;
The interface of the backup cluster data service unit;
The data file information that the backup cluster data service unit is stored;
The database engine serial number of the backup cluster data service unit;
The database engine serial number of the alternate device of the backup cluster data service unit.
Further, the synchronizer also includes:
Full dose data synchronization unit, for obtaining the information of newly-built backup cluster data storage device, And based on the information, from the main company-data service equipment snapshot data is obtained, and by the snapshot Data is activation is to the data service unit of newly-built backup cluster synchronizing.
Further, the synchronizer also includes:
Broadcaster is updated, for maintaining the number with other synchronizers by fixed time broadcast data mechanism According to synchronized update.
Further, the data logging includes data operation information, data key and data value.
Compared with prior art, according to the embodiment of the present application, by obtaining from main cluster configuration service equipment Distribution list is taken, the distribution list includes the letter of the main company-data service equipment for needing to safeguard synchronous Breath;Based on the distribution list, data logging is obtained from the main company-data service equipment, and Synchronous task information is generated based on the data logging;The synchronous task information is sent to backup set Group's data service unit is delayed machine or even main collection with synchronizing so as to work as main collection data service device Group is overall delay machine when, the data service unit that can continue as taking over data before machine of synchronously delaying take The particular state of business equipment, so as to avoid dirty data or loss of data.
Description of the drawings
By reading the detailed description made to non-limiting example made with reference to the following drawings, this Shen Other features, objects and advantages please will become more apparent upon:
Fig. 1 illustrates the synchronization for distributed cache system data syn-chronization according to the application one side Equipment schematic diagram;
Fig. 2 illustrates the synchronizer and main cluster and Backup Data according to one preferred embodiment of the application Cooperation schematic diagram;
Fig. 3 is illustrated according to the application another aspect for distributed cache system method of data synchronization Schematic flow sheet;
Fig. 4 illustrates the synchronizer and main cluster and Backup Data according to one preferred embodiment of the application Cooperation schematic flow sheet.
Same or analogous reference represents same or analogous part in accompanying drawing.
Specific embodiment
The application is described in further detail below in conjunction with the accompanying drawings.
Distributed cache system generally includes two or more cluster, and each cluster is matched somebody with somebody including at least one Put service equipment (Config Server) and some data service units (Data Sever), configuration clothes Business equipment is responsible for all of data service unit, and safeguards the status information number of data service unit According to data service unit externally provides various data, services, and is converged self-condition in the form of heart beating Offer configuration service equipment, configuration service equipment is control point, generally using one lead a standby form come Ensure its reliability.
Configuration service equipment the key (Key) of data is assigned in some buckets (bucket be load balancing and The ultimate unit of Data Migration), and each barrel is assigned to by different data storages based on certain strategy In equipment, with equilibrium criterion distribution.When there is certain data storage device to break down unavailable, match somebody with somebody Putting service equipment can find this situation, and be responsible for recalculating each new bucket in data, services The distribution table of equipment, by the access of the bucket of failed machines service will be reassigned into other data originally In service equipment.This when, it may happen that the migration of data.Adjustment route after the completion of migration, New configuration information can be all pushed to data service unit by the change of route every time, configuration service equipment.
In two-shipper house-owner backup cluster (double the to hold up two-shipper room) deployment way of distributed cache system, lead to Often include a main cluster and one or more backup clusters, main cluster and backup cluster are respectively at two In machine room.Each cluster (main cluster or backup cluster) includes at least one configuration service equipment (Config Server) and some data service units (Data Sever), under normal circumstances, use Family only uses main cluster, read-write data all to interact with main cluster, and backup cluster needs to carry out data syn-chronization Process, to keep keeping data syn-chronization with main cluster, when main cluster breaks down, backs up cluster meeting Cluster provides service based on switching to, and after former main cluster recovery is normal, then switches back, to ensure The clock availability of system.
In two-shipper house-owner backup cluster deployment way, one can be configured and taken independently of main cluster configuration The equipment for carrying out data syn-chronization of business equipment, for managing synchrodata, including will actively treat same Step data is pushed in requisition for synchronous backup cluster data service unit and the backup cluster number According to the alternate device of service equipment, then main cluster configuration equipment need to record all synchronizing processes and Connection status, when main collection data service device delay machine in addition main cluster integrally delay machine when, can Synchronously delayed the particular state of the data service unit before machine with continuing as the data service unit taken over, from And avoid dirty data or loss of data.
Fig. 1 illustrates the synchronization for distributed cache system data syn-chronization according to the application one side Equipment schematic diagram, the synchronizer 1 includes:Distribution list acquisition device 11, data logging is obtained Device 12 and synchronizer 13.
The distribution list acquisition device 11 obtains distribution list, institute from main cluster configuration service equipment State the information of the main company-data service equipment that distribution list includes needing to safeguard synchronous;The data day Will acquisition device 12 is based on the distribution list, and from the main company-data service equipment number is obtained According to daily record, and synchronous task information is generated based on the data logging;The synchronizer 13 is by institute State synchronous task information to send to backup cluster data service unit to synchronize.
Here, the synchronizer 1 can be by network host, single network server, multiple networks clothes Cloud that business device collection or multiple servers are constituted etc. is realized.Here, cloud is by based on cloud computing (Cloud Computing a large amount of main frames or the webserver) are constituted, wherein, cloud computing is Distributed Calculation One kind, a super virtual computer being made up of the loosely-coupled computer collection of a group.This area Technical staff will be understood that the above-mentioned network equipment is only for example, and other are existing or are likely to occur from now on The network equipment is such as applicable to the application, also should be included within the application protection domain, and here with Way of reference is incorporated herein.Here, the network host according to being previously set or can be deposited including one kind The instruction of storage, carries out the electronic equipment of numerical computations and information processing automatically, and its hardware includes but do not limit In microprocessor, special IC (ASIC), programmable gate array (FPGA), digital processing unit (DSP), embedded device etc..
Those skilled in the art will be understood that above-mentioned synchronizer 1 is only for example, other existing or the presents The synchronizer 1 being likely to occur afterwards is such as applicable to the application, should also be included in the application protection domain Within, and here is incorporated herein by reference.
Here, the data logging includes data operation information, data key and data value.
Fig. 2 illustrates the synchronizer and main cluster and Backup Data according to one preferred embodiment of the application Cooperation schematic diagram, below in conjunction with Fig. 1 and Fig. 2, describe the course of work of synchronizer 1 in detail.
The distribution list acquisition device 11 can be from the database engine (example of main cluster configuration service equipment Such as leveldb) (data logging is the operation day of database engine itself to the local data logging of middle reading Will, such as binlog, binary log file), synchronous task information is formed after parsing, it is stored in task Queue (binlog queue), then needs synchronization is sent to by synchronizer (such as client) Backup cluster.
Herein described synchronizer 1 can obtain data processing daily record from main cluster configuration service equipment File, and actively stored data to be synchronized from main company-data based on the data processing journal file Equipment is pulled out, and is sent in requisition for synchronous backup cluster data storage device, the synchronizer 1, independently of main cluster or backup cluster, is capable of achieving separate configurations, can flexibly use, and can avoid Main cluster delay machine when dirty data or data loss.
The distribution list acquisition device 11 obtains distribution list, institute from main cluster configuration service equipment State the information of the main company-data service equipment that distribution list includes needing to safeguard synchronous.Specifically, institute State distribution list acquisition device 11 to maintain and the connection of the main cluster configuration equipment by heartbeat mechanism, And the heart beating based on the main cluster configuration equipment is fed back, the distribution list is obtained.
The data logging acquisition device 12 is based on the distribution list, takes from the main company-data Data logging is obtained in business equipment, and synchronous task information is generated based on the data logging.
Specifically, the data logging acquisition device 12 is based on the distribution list, obtains the master The link information of company-data service equipment;Based on the link information, take with the main company-data Business equipment sets up connection, and obtains data logging.
Specifically, the data logging acquisition device 12 is based on the distribution list, from the main collection Group configuration service equipment obtains the link information of the main company-data service equipment, or lasting from The link information of main company-data service equipment is obtained in the data for changing storage.
Here, the link information of the main company-data service equipment includes at least following any one:Institute State net association (IP) of main company-data service equipment;The interface of the main company-data service equipment (Port)。
According to the net association of the data service unit of backup cluster, interface connection, then |input paramete data day The file name (binlog filename) of will, document location (Position), main company-data storage Equipment reads data logging, and the data frame sign of data logging is transferred to into synchronizer 1, synchronous Equipment 1 is sent to backup cluster after data logging is parsed, to complete synchronization.
Preferably, the synchronizer 1 also includes:Configuration information acquisition device, for from the master Cluster configuration service equipment obtains the configuration information of the main company-data service equipment, and based on described Configuration information, newly-built or renewal has the data of persistent storage;Wherein, the main company-data The configuration information of service equipment includes:The net association of the main company-data service equipment;The main cluster The interface of data service unit;The data file information that the main company-data service equipment is stored; The database engine serial number of the main company-data service equipment;The main company-data service equipment Alternate device database engine serial number.
Here, the main company-data service equipment periodically can obtain alternate device from alternate device Database engine serial number is simultaneously broadcasted, and synchronizer 1 can obtain the data base of respective backup equipment Engine serial number.
After a certain main company-data service equipment delays machine, taken over by the alternate device of main cluster, it is described Synchronizer 1 needs the synchronous regime of the alternate device for knowing the main cluster before the machine of delaying, described with confidence Configuration information acquired in breath acquisition device includes the data base of the main company-data service equipment The database engine serial number of the alternate device of engine serial number and the main company-data service equipment, By the mapping corresponding relation of two serial numbers, the synchronous shape of respective backup equipment in alternate device is found State, to continue the machine of delaying before data syn-chronization operation.Now, the synchronizer 1 is from main cluster configuration The equipment of data, services is provided in the configuration information that service equipment is obtained accordingly can be changed, then can be with According to the database engine serial number and the main company-data clothes of the main company-data service equipment The corresponding relation of the database engine serial number of the alternate device of business equipment, finds the corresponding data that provide and takes The alternate device of business, then obtain data logging from alternate device.
The synchronizer 13 sends the synchronous task information to backup cluster data service unit To synchronize.Specifically, the synchronizer is used for:Obtain from backup cluster configuration service equipment Take the configuration information of the backup cluster data service unit;Set based on the backup cluster data, services Standby configuration information, the synchronous task information is sent to the corresponding data service equipment of backup cluster To synchronize.The configuration information of the backup cluster data service unit includes at least following arbitrary :The net association of the backup cluster data service unit;It is described to back up connecing for cluster data service unit Mouthful;The data file information that the backup cluster data service unit is stored;The backup cluster number According to the database engine serial number of service equipment;The alternate device of the backup cluster data service unit Database engine serial number.
The synchronizer 1 also includes:Full dose data synchronization unit, for obtaining newly-built backup set The information of group's data storage device, and based on the information, obtain from the main company-data service equipment Snapshot data, and by the snapshot data send data service unit to newly-built backup cluster with Synchronize.If midway creates backup cluster, can the synchronous main company-data service equipment of first full dose Snapshot data after, increment synchronization is carried out using default position.
The synchronizer 1 also includes:Broadcaster is updated, for by fixed time broadcast data mechanism Maintain the data synchronization updating with other synchronizers.
According to the synchronization system that the preferred embodiment of the application one is provided, synchronization system sets including some synchronizations Standby 1, before activation single-point is registered in main cluster configuration service equipment, to match somebody with somebody each synchronizer 1 Put and service, main cluster configuration service equipment draws distribution according to the concordance hash algorithm of load balancing List (rsync task tables), each synchronizer 1 maintains heart beating with the main cluster configuration equipment, To obtain distribution list, the distribution list maintains the corresponding needs of each synchronizer 1 and safeguards same The some main company-data storage device of step, main cluster configuration equipment can be according to configured synchronizer 1 and the database engine of existing main company-data storage device generate the distribution list.Described point Generating with list can distribute according to the concordance hash algorithm of load balancing.Distribution list every time is generated All carry version number.Meanwhile, each synchronizer 1 is wide by timing using broadcaster is updated Itself current state is broadcast to other synchronizers 1 by the timing of multicast data mechanism, is maintained synchronous with other The data synchronization updating of equipment, the dynamics of fixed time broadcast is little, when a certain synchronizer 1 delays machine, Existing synchronizer 1 is redistributed, simultaneously operating can be proceeded.
Fig. 3 is illustrated according to the application another aspect for distributed cache system method of data synchronization Schematic flow sheet, methods described includes:Step S11, step S12 and step S13.
Step S11 includes:Distribution list, the distribution are obtained from main cluster configuration service equipment List includes the information of the main company-data service equipment for needing to safeguard synchronous;The step 12 includes: Based on the distribution list, data logging is obtained from the main company-data service equipment, and be based on The data logging generates synchronous task information;Step S13 includes:By synchronous task letter Breath is sent to backup cluster data service unit to synchronize.
Here, the synchronizer 1 can be by network host, single network server, multiple networks clothes Cloud that business device collection or multiple servers are constituted etc. is realized.Here, cloud is by based on cloud computing (Cloud Computing a large amount of main frames or the webserver) are constituted, wherein, cloud computing is Distributed Calculation One kind, a super virtual computer being made up of the loosely-coupled computer collection of a group.This area Technical staff will be understood that the above-mentioned network equipment is only for example, and other are existing or are likely to occur from now on The network equipment is such as applicable to the application, also should be included within the application protection domain, and here with Way of reference is incorporated herein.Here, the network host according to being previously set or can be deposited including one kind The instruction of storage, carries out the electronic equipment of numerical computations and information processing automatically, and its hardware includes but do not limit In microprocessor, special IC (ASIC), programmable gate array (FPGA), digital processing unit (DSP), embedded device etc..
Those skilled in the art will be understood that above-mentioned synchronizer 1 is only for example, other existing or the presents The synchronizer 1 being likely to occur afterwards is such as applicable to the application, should also be included in the application protection domain Within, and here is incorporated herein by reference.
Here, the data logging includes data operation information, data key and data value.
Fig. 4 illustrates the synchronizer and main cluster and Backup Data according to one preferred embodiment of the application Cooperation schematic diagram, below in conjunction with Fig. 3 and Fig. 4, describe the course of work of methods described in detail.
Can be from the database engine of main cluster configuration service equipment (for example in step S11 Leveldb read in) local data logging (data logging for database engine itself Operation Log, Such as binlog, binary log file), synchronous task information is formed after parsing, it is stored in task queue (binlog queue), then the backup for needing synchronization is sent to by synchronizer (such as client) Cluster.
Herein described method can obtain data processing journal file from main cluster configuration service equipment, And based on the data processing journal file actively by data to be synchronized from main company-data storage device Pull out, send in requisition for synchronous backup cluster data storage device, methods described is independently of master Cluster or backup cluster, are capable of achieving separate configurations, can flexibly use, and main cluster can be avoided to occur Delay machine when dirty data or data loss.
In step S11, from main cluster configuration service equipment distribution list, the distribution are obtained List includes the information of the main company-data service equipment for needing to safeguard synchronous.Specifically, the step 11 maintain the connection with the main cluster configuration equipment by heartbeat mechanism, and based on the main collection flock mating Standby heart beating feedback is installed, the distribution list is obtained.
In step S12, based on the distribution list, from the main company-data service equipment Middle acquisition data logging, and synchronous task information is generated based on the data logging.
Specifically, the step 12 is based on the distribution list, obtains the main company-data service The link information of equipment;Based on the link information, set up with the main company-data service equipment and connect Connect, and obtain data logging.
Specifically, the step 12 is based on the distribution list, takes from the main cluster configuration Business equipment obtains the link information of the main company-data service equipment, or from existing persistent storage The link information of main company-data service equipment is obtained in data.
Here, the link information of the main company-data service equipment includes at least following any one:Institute State net association (IP) of main company-data service equipment;The interface of the main company-data service equipment (Port)。
According to the net association of the data service unit of backup cluster, interface connection, then |input paramete data day The file name (binlog filename) of will, document location (Position), main company-data storage Equipment reads data logging, and obtains the data frame sign transmission of data logging, then by data logging solution Backup cluster is sent to after analysis, to complete synchronization.
Preferably, methods described also includes:Configuration information acquisition device, for from the main collection flock mating The configuration information that service equipment obtains the main company-data service equipment is put, and based on described with confidence Breath, newly-built or renewal has the data of persistent storage;Wherein, the main company-data service sets Standby configuration information includes:The net association of the main company-data service equipment;The main company-data clothes The interface of business equipment;The data file information that the main company-data service equipment is stored;The master The database engine serial number of company-data service equipment;The backup of the main company-data service equipment The database engine serial number of equipment.
Here, the main company-data service equipment periodically can obtain alternate device from alternate device Database engine serial number is simultaneously broadcasted, and methods described can obtain the data base of respective backup equipment and draw Hold up serial number.
After a certain main company-data service equipment delays machine, taken over by the alternate device of main cluster, it is described Method needs the synchronous regime of the alternate device for knowing the main cluster before the machine of delaying, the configuration information to obtain Configuration information acquired in device includes the database engine sequence of the main company-data service equipment The database engine serial number of the alternate device of row number and the main company-data service equipment, by two The mapping corresponding relation of individual serial number, finds the synchronous regime of respective backup equipment in alternate device, with Continue the operation of the data syn-chronization before the machine of delaying.Now, methods described is obtained from main cluster configuration service equipment Configuration information in provide data, services equipment accordingly can change, then can be according to the main collection The database engine serial number of group's data service unit and the backup of the main company-data service equipment The corresponding relation of the database engine serial number of equipment, finds the corresponding backup for providing data, services and sets It is standby, then obtain data logging from alternate device.
Step S13 by the synchronous task information send to backup cluster data service unit with Synchronize.Specifically, the synchronizer is used for:Obtain from backup cluster configuration service equipment The configuration information of the backup cluster data service unit;Based on the backup cluster data service unit Configuration information, by the synchronous task information send to backup cluster corresponding data service equipment with Synchronize.The configuration information of the backup cluster data service unit includes at least following any one: The net association of the backup cluster data service unit;The interface of the backup cluster data service unit; The data file information that the backup cluster data service unit is stored;The backup company-data clothes The database engine serial number of business equipment;The number of the alternate device of the backup cluster data service unit According to storehouse engine serial number.
Methods described also includes:The information of newly-built backup cluster data storage device is obtained, and is based on The information, from the main company-data service equipment snapshot data is obtained, and the snapshot data is sent out The data service unit of newly-built backup cluster is delivered to synchronize.If midway creates backup Cluster, can first full dose synchronously after the snapshot data of main company-data service equipment, using default position Carry out increment synchronization.
Methods described also includes:Data with other synchronizers are maintained by fixed time broadcast data mechanism Synchronized update.
Compared with prior art, according to the embodiment of the present application, by obtaining from main cluster configuration service equipment Distribution list is taken, the distribution list includes the letter of the main company-data service equipment for needing to safeguard synchronous Breath;Based on the distribution list, data logging is obtained from the main company-data service equipment, and Synchronous task information is generated based on the data logging;The synchronous task information is sent to backup set Group's data service unit is delayed machine or even main collection with synchronizing so as to work as main collection data service device Group is overall delay machine when, the data service unit that can continue as taking over data before machine of synchronously delaying take The particular state of business equipment, so as to avoid dirty data or loss of data.
It should be noted that the application can be carried out in the assembly of software and/or software with hardware, example Such as, can be set using special IC (ASIC), general purpose computer or any other similar hardware It is standby realizing.In one embodiment, the software program of the application can pass through computing device to realize Steps described above or function.Similarly, the software program (including related data structure) of the application Can be stored in computer readable recording medium storing program for performing, for example, RAM memory, magnetically or optically driver or Floppy disc and similar devices.In addition, some steps or function of the application can employ hardware to realize, example Such as, as coordinating so as to perform the circuit of each step or function with processor.
In addition, the part of the application can be applied to computer program, such as computer journey Sequence is instructed, and when it is computer-executed, by the operation of the computer, can be called or be provided According to the present processes and/or technical scheme.And the programmed instruction of the present processes is called, can During fixed or moveable recording medium can be stored in, and/or held by broadcast or other signals Carry the data flow in media and be transmitted, and/or be stored in the meter according to described program instruction operation In calculating the working storage of machine equipment.Here, according to one embodiment of the application including a dress Put, the device includes the memorizer for storing computer program instructions and for execute program instructions Processor, wherein, when the computer program instructions are by the computing device, trigger the device Methods and/or techniques scheme of the operation based on aforementioned multiple embodiments according to the application.
It is obvious to a person skilled in the art that the application is not limited to the thin of above-mentioned one exemplary embodiment Section, and in the case of without departing substantially from spirit herein or basic feature, can be with other concrete Form realizes the application.Therefore, no matter from the point of view of which point, embodiment all should be regarded as exemplary , and be nonrestrictive, scope of the present application is by claims rather than described above is limited It is fixed, it is intended that all changes in the implication and scope of the equivalency of claim that will fall are included In the application.Any reference in claim should not be considered as into the right involved by limiting will Ask.Furthermore, it is to be understood that " an including " word is not excluded for other units or step, odd number is not excluded for plural number.Dress Putting multiple units or device of statement in claim can also pass through software by a unit or device Or hardware is realizing.The first, the second grade word is used for representing title, and is not offered as any specific Order.

Claims (22)

1. a kind of method for realizing distributed cache system data syn-chronization, wherein, methods described includes:
Distribution list is obtained from main cluster configuration service equipment, the distribution list includes needing to safeguard same The information of the main company-data service equipment of step;
Based on the distribution list, data logging is obtained from the main company-data service equipment, and Synchronous task information is generated based on the data logging;
The synchronous task information is sent to backup cluster data service unit to synchronize.
2. method according to claim 1, wherein, obtain point from main cluster configuration service equipment Include with list:
Connection with the main cluster configuration equipment is maintained by heartbeat mechanism;
Heart beating based on the main cluster configuration equipment is fed back, and obtains the distribution list.
3. method according to claim 1 and 2, wherein, based on the distribution list, from institute Stating acquisition data logging in main company-data service equipment includes:
Based on the distribution list, the link information of the main company-data service equipment is obtained;
Based on the link information, set up with the main company-data service equipment and be connected, and obtain number According to daily record.
4. method according to claim 3, wherein, obtain the main company-data service equipment Link information include:
Based on the distribution list, from the main cluster configuration service equipment the main company-data is obtained The link information of service equipment, or main company-data service is obtained from the data of existing persistent storage The link information of equipment.
5. the method according to claim 3 or 4, wherein, the main company-data service equipment Link information include at least following any one:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment.
6. method according to any one of claim 1 to 5, wherein, methods described also includes:
From the main cluster configuration service equipment obtain the main company-data service equipment with confidence Breath;
Based on the configuration information, newly-built or renewal has the data of persistent storage;
Wherein, the configuration information of the main company-data service equipment includes:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment;
The data file information that the main company-data service equipment is stored;
The database engine serial number of the main company-data service equipment;
The database engine serial number of the alternate device of the main company-data service equipment.
7. method according to any one of claim 1 to 6, wherein, by the synchronous task Information send to backup cluster data service unit with synchronize including:
The configuration of the backup cluster data service unit is obtained from backup cluster configuration service equipment Information;
Based on the configuration information of the backup cluster data service unit, the synchronous task information is sent out The corresponding data service equipment of backup cluster is delivered to synchronize.
8. method according to claim 7, wherein, the backup cluster data service unit Configuration information includes at least following any one:
The net association of the backup cluster data service unit;
The interface of the backup cluster data service unit;
The data file information that the backup cluster data service unit is stored;
The database engine serial number of the backup cluster data service unit;
The database engine serial number of the alternate device of the backup cluster data service unit.
9. the method according to any one of claim 1 to 8, wherein, methods described also includes:
The information of newly-built backup cluster data storage device is obtained, and based on the information, from the master Company-data service equipment obtains snapshot data;
The snapshot data is sent into same to carry out to the data service unit of newly-built backup cluster Step.
10. method according to any one of claim 1 to 9, wherein, methods described also includes:
Data synchronization updating with other synchronizers is maintained by fixed time broadcast data mechanism.
11. methods according to any one of claim 1 to 10, wherein, the data logging Including data operation information, data key and data value.
A kind of 12. synchronizers for realizing distributed cache system data syn-chronization, wherein, the synchronization sets It is standby to include:
Distribution list acquisition device, it is described for obtaining distribution list from main cluster configuration service equipment Distribution list includes the information of the main company-data service equipment for needing to safeguard synchronous;
Data logging acquisition device, for based on the distribution list, from the main company-data service Data logging is obtained in equipment, and synchronous task information is generated based on the data logging;
Synchronizer, for by the synchronous task information send to backup cluster data service unit with Synchronize.
13. synchronizers according to claim 12, wherein, the distribution list acquisition device For:
Connection with the main cluster configuration equipment is maintained by heartbeat mechanism;
Heart beating based on the main cluster configuration equipment is fed back, and obtains the distribution list.
14. synchronizers according to claim 10 or 11, wherein, the data logging is obtained Device is used for:
Based on the distribution list, the link information of the main company-data service equipment is obtained;
Based on the link information, set up with the main company-data service equipment and be connected, and obtain number According to daily record.
15. synchronizers according to claim 14, wherein, the data logging acquisition device For:
Based on the distribution list, from the main cluster configuration service equipment the main company-data is obtained The link information of service equipment, or main company-data service is obtained from the data of existing persistent storage The link information of equipment.
16. synchronizers according to claims 14 or 15, wherein, the main company-data clothes The link information of business equipment includes at least following any one:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment.
17. synchronizers with any one of according to claim 12 to 16, wherein, the synchronization Equipment also includes:
Configuration information acquisition device, for obtaining the main cluster from the main cluster configuration service equipment The configuration information of data service unit, and based on the configuration information, newly-built or renewal has persistently Change the data of storage;
Wherein, the configuration information of the main company-data service equipment includes:
The net association of the main company-data service equipment;
The interface of the main company-data service equipment;
The data file information that the main company-data service equipment is stored;
The database engine serial number of the main company-data service equipment;
The database engine serial number of the alternate device of the main company-data service equipment.
18. synchronizers according to any one of claim 12 to 17, wherein, the synchronization Device is used for:
The configuration of the backup cluster data service unit is obtained from backup cluster configuration service equipment Information;
Based on the configuration information of the backup cluster data service unit, the synchronous task information is sent out The corresponding data service equipment of backup cluster is delivered to synchronize.
19. synchronizers according to claim 18, wherein, the backup cluster data, services The configuration information of equipment includes at least following any one:
The net association of the backup cluster data service unit;
The interface of the backup cluster data service unit;
The data file information that the backup cluster data service unit is stored;
The database engine serial number of the backup cluster data service unit;
The database engine serial number of the alternate device of the backup cluster data service unit.
20. synchronizers according to any one of claim 12 to 19, wherein, the synchronization Equipment also includes:
Full dose data synchronization unit, for obtaining the information of newly-built backup cluster data storage device, And based on the information, from the main company-data service equipment snapshot data is obtained, and by the snapshot Data is activation is to the data service unit of newly-built backup cluster synchronizing.
21. synchronizers according to any one of claim 12 to 20, wherein, the synchronization Equipment also includes:
Broadcaster is updated, for maintaining the number with other synchronizers by fixed time broadcast data mechanism According to synchronized update.
22. synchronizers according to any one of claim 12 to 21, wherein, the data Daily record includes data operation information, data key and data value.
CN201510648329.3A 2015-10-09 2015-10-09 Method and equipment for data synchronization of distributed caching system Pending CN106570007A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510648329.3A CN106570007A (en) 2015-10-09 2015-10-09 Method and equipment for data synchronization of distributed caching system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510648329.3A CN106570007A (en) 2015-10-09 2015-10-09 Method and equipment for data synchronization of distributed caching system

Publications (1)

Publication Number Publication Date
CN106570007A true CN106570007A (en) 2017-04-19

Family

ID=58507526

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510648329.3A Pending CN106570007A (en) 2015-10-09 2015-10-09 Method and equipment for data synchronization of distributed caching system

Country Status (1)

Country Link
CN (1) CN106570007A (en)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107729176A (en) * 2017-09-14 2018-02-23 北京京东尚科信息技术有限公司 The disaster recovery method and disaster tolerance system of a kind of profile management systems
CN108121559A (en) * 2017-12-29 2018-06-05 重庆金融资产交易所有限责任公司 Configuration file method for pushing, server and storage medium
CN108322533A (en) * 2018-01-31 2018-07-24 广州鼎甲计算机科技有限公司 Configuration and synchronization method between distributed type assemblies node based on operation log
CN108418859A (en) * 2018-01-24 2018-08-17 华为技术有限公司 The method and apparatus for writing data
CN108415951A (en) * 2018-02-02 2018-08-17 广东睿江云计算股份有限公司 A kind of database control method and system
CN109189860A (en) * 2018-10-19 2019-01-11 山东浪潮云信息技术有限公司 A kind of active and standby increment synchronization method of MySQL based on Kubernetes system
CN109408280A (en) * 2017-08-17 2019-03-01 北京金山云网络技术有限公司 Data back up method, apparatus and system
CN109739690A (en) * 2018-12-29 2019-05-10 平安科技(深圳)有限公司 Backup method and Related product
WO2019091324A1 (en) * 2017-11-07 2019-05-16 阿里巴巴集团控股有限公司 Data synchronization method and device, and electronic device
CN109871295A (en) * 2017-12-01 2019-06-11 北京金山云网络技术有限公司 A kind of data back up method, back-up device, electronic equipment and storage medium
CN110377577A (en) * 2018-04-11 2019-10-25 北京嘀嘀无限科技发展有限公司 Method of data synchronization, device, system and computer readable storage medium
CN110635953A (en) * 2019-10-17 2019-12-31 厦门网宿有限公司 Configuration information management method and device
CN112527567A (en) * 2020-12-24 2021-03-19 北京百度网讯科技有限公司 System disaster tolerance method, device, equipment and storage medium
CN112988882A (en) * 2019-12-12 2021-06-18 阿里巴巴集团控股有限公司 System, method and device for data remote disaster recovery and computing equipment
CN113312384A (en) * 2020-02-26 2021-08-27 阿里巴巴集团控股有限公司 Graph data query processing method and device and electronic equipment
CN114285865A (en) * 2021-12-28 2022-04-05 天翼云科技有限公司 Access authority control system for sharing cloud hard disk
CN114925059A (en) * 2022-07-20 2022-08-19 阿里巴巴达摩院(杭州)科技有限公司 Dirty data processing method, core network, device and storage medium
WO2023046042A1 (en) * 2021-09-23 2023-03-30 华为技术有限公司 Data backup method and database cluster

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346779A (en) * 2011-10-18 2012-02-08 中国联合网络通信集团有限公司 Distributed file system and master control node backup method
CN102693324A (en) * 2012-01-09 2012-09-26 西安电子科技大学 Distributed database synchronization system, synchronization method and node management method
CN104239476A (en) * 2014-09-04 2014-12-24 上海天脉聚源文化传媒有限公司 Method, device and system for synchronizing databases

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102346779A (en) * 2011-10-18 2012-02-08 中国联合网络通信集团有限公司 Distributed file system and master control node backup method
CN102693324A (en) * 2012-01-09 2012-09-26 西安电子科技大学 Distributed database synchronization system, synchronization method and node management method
CN104239476A (en) * 2014-09-04 2014-12-24 上海天脉聚源文化传媒有限公司 Method, device and system for synchronizing databases

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109408280A (en) * 2017-08-17 2019-03-01 北京金山云网络技术有限公司 Data back up method, apparatus and system
CN107729176A (en) * 2017-09-14 2018-02-23 北京京东尚科信息技术有限公司 The disaster recovery method and disaster tolerance system of a kind of profile management systems
CN107729176B (en) * 2017-09-14 2020-09-29 北京京东尚科信息技术有限公司 Disaster recovery method and disaster recovery system for configuration file management system
CN110019514B (en) * 2017-11-07 2023-05-09 阿里巴巴集团控股有限公司 Data synchronization method and device and electronic equipment
CN110019514A (en) * 2017-11-07 2019-07-16 阿里巴巴集团控股有限公司 Method of data synchronization, device and electronic equipment
WO2019091324A1 (en) * 2017-11-07 2019-05-16 阿里巴巴集团控股有限公司 Data synchronization method and device, and electronic device
CN109871295A (en) * 2017-12-01 2019-06-11 北京金山云网络技术有限公司 A kind of data back up method, back-up device, electronic equipment and storage medium
CN109871295B (en) * 2017-12-01 2022-04-05 北京金山云网络技术有限公司 Data backup method, backup device, electronic equipment and storage medium
CN108121559A (en) * 2017-12-29 2018-06-05 重庆金融资产交易所有限责任公司 Configuration file method for pushing, server and storage medium
CN108418859B (en) * 2018-01-24 2020-11-06 华为技术有限公司 Method and device for writing data
CN108418859A (en) * 2018-01-24 2018-08-17 华为技术有限公司 The method and apparatus for writing data
CN108322533A (en) * 2018-01-31 2018-07-24 广州鼎甲计算机科技有限公司 Configuration and synchronization method between distributed type assemblies node based on operation log
CN108415951A (en) * 2018-02-02 2018-08-17 广东睿江云计算股份有限公司 A kind of database control method and system
CN108415951B (en) * 2018-02-02 2022-01-11 广东睿江云计算股份有限公司 Database control method and system
CN110377577B (en) * 2018-04-11 2022-03-04 北京嘀嘀无限科技发展有限公司 Data synchronization method, device, system and computer readable storage medium
CN110377577A (en) * 2018-04-11 2019-10-25 北京嘀嘀无限科技发展有限公司 Method of data synchronization, device, system and computer readable storage medium
CN109189860A (en) * 2018-10-19 2019-01-11 山东浪潮云信息技术有限公司 A kind of active and standby increment synchronization method of MySQL based on Kubernetes system
CN109739690B (en) * 2018-12-29 2024-05-28 平安科技(深圳)有限公司 Backup method and related products
CN109739690A (en) * 2018-12-29 2019-05-10 平安科技(深圳)有限公司 Backup method and Related product
WO2021072967A1 (en) * 2019-10-17 2021-04-22 厦门网宿有限公司 Method and device for managing configuration information
CN110635953A (en) * 2019-10-17 2019-12-31 厦门网宿有限公司 Configuration information management method and device
CN112988882A (en) * 2019-12-12 2021-06-18 阿里巴巴集团控股有限公司 System, method and device for data remote disaster recovery and computing equipment
CN112988882B (en) * 2019-12-12 2024-01-23 阿里巴巴集团控股有限公司 System, method and device for preparing data from different places and computing equipment
CN113312384A (en) * 2020-02-26 2021-08-27 阿里巴巴集团控股有限公司 Graph data query processing method and device and electronic equipment
CN113312384B (en) * 2020-02-26 2023-12-26 阿里巴巴集团控股有限公司 Query processing method and device for graph data and electronic equipment
CN112527567A (en) * 2020-12-24 2021-03-19 北京百度网讯科技有限公司 System disaster tolerance method, device, equipment and storage medium
WO2023046042A1 (en) * 2021-09-23 2023-03-30 华为技术有限公司 Data backup method and database cluster
CN114285865B (en) * 2021-12-28 2023-08-08 天翼云科技有限公司 Access authority control system for shared cloud hard disk
CN114285865A (en) * 2021-12-28 2022-04-05 天翼云科技有限公司 Access authority control system for sharing cloud hard disk
CN114925059A (en) * 2022-07-20 2022-08-19 阿里巴巴达摩院(杭州)科技有限公司 Dirty data processing method, core network, device and storage medium

Similar Documents

Publication Publication Date Title
CN106570007A (en) Method and equipment for data synchronization of distributed caching system
US11770447B2 (en) Managing high-availability file servers
US20160105502A1 (en) Data synchronization method, data synchronization apparatus, and distributed system
Akkoorath et al. Cure: Strong semantics meets high availability and low latency
US10209979B2 (en) System and method for distributed revision control
CN105493474B (en) System and method for supporting partition level logging for synchronizing data in a distributed data grid
US8930316B2 (en) System and method for providing partition persistent state consistency in a distributed data grid
US9235623B2 (en) Policy-based storage structure distribution
US10430217B2 (en) High availability using dynamic quorum-based arbitration
CN104378423B (en) Metadata cluster distributed memory system and reading, the method for write-in storage data
KR101670343B1 (en) Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching
US8751641B2 (en) Optimizing clustered network attached storage (NAS) usage
US20100023564A1 (en) Synchronous replication for fault tolerance
JP2008059583A (en) Cluster system, method for backing up replica in cluster system, and program product
CN106528574A (en) Data synchronization method and device
AU2015241457A1 (en) Geographically-distributed file system using coordinated namespace replication
CN106055698A (en) Data migration method, agent node and database instance
EP3080697A1 (en) System and method for supporting persistence partition recovery in a distributed data grid
CN113268472B (en) Distributed data storage system and method
CN105610947A (en) Method, device and system for realizing high-available distributed queue service
US20140101110A1 (en) High availability event log collection in a networked system
US10152493B1 (en) Dynamic ephemeral point-in-time snapshots for consistent reads to HDFS clients
EP3039568B1 (en) Distributed disaster recovery file sync server system
Kim et al. A distributed NameNode cluster for a highly-available Hadoop distributed file system
CN115967611A (en) Cross-domain switching processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20170419