CN103023879B - Wide-area data synchronous method between a kind of data center based on high-speed cache - Google Patents
Wide-area data synchronous method between a kind of data center based on high-speed cache Download PDFInfo
- Publication number
- CN103023879B CN103023879B CN201210486206.0A CN201210486206A CN103023879B CN 103023879 B CN103023879 B CN 103023879B CN 201210486206 A CN201210486206 A CN 201210486206A CN 103023879 B CN103023879 B CN 103023879B
- Authority
- CN
- China
- Prior art keywords
- data
- data center
- synchronous
- remote
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses wide-area data synchronous method between a kind of data center based on high-speed cache, it is characterized in that comprising the following steps: the initialization of (1) synchro system; The server of local data center according to configuring system access data graftabl frequently, and communicates with needing each synchronous remote data center, and then the difference of comparing data carry out synchronous; (2) read and write access of data; Buffer memory in all directly internally being deposited by interface the access of data is read and write, and the service routine on backstage has one or more separate threads by the data write hard disk in internal memory; (3) remote synchronization of data; Periodically the content that interface interchange is revised directly is synchronized to remote data center from internal memory, remote data center performs same interface interchange and namely completes data syn-chronization.The present invention has that synchronous efficiency is high, occupied bandwidth is little, reliability is high, portability is good, dispose simple advantage, other abnormal inconsistence problems produced when solving system cloud gray model.
Description
Technical field
The present invention relates to wide-area data synchronous method between a kind of data center based on high-speed cache, belong to communication technical field.
Background technology
Existing real time data management system, is generally in units of a data center, carries out the distributed management of data.Even if there is the synchronizing function of remote data center, also have lock in time long, the shortcomings such as bandwidth occupancy is large, and often system is very complicated, the input cost brought to user is large.
Summary of the invention
For solving the deficiency in prior art, the present invention seeks to be to be to provide that a kind of synchronous efficiency is high, occupied bandwidth is little, reliability is high, portability is good, dispose wide-area data synchronous method between simple data center.
For achieving the above object, technical scheme of the present invention is:
Wide-area data synchronous method between a kind of data center based on high-speed cache.The present invention with remote data center synchronously for main purpose, improving local data access efficiency is secondary objective, with incremental update, increment synchronization and file backup for method, provide a kind of local and remote access efficiency high, the data volume of synchronous transmission is little, wide-area data synchronous method between the data center that also can ensure system conformance under the condition of remote server off-line a period of time.
The present invention adopts following technical scheme to realize:
Wide-area data synchronous method between a kind of data center based on high-speed cache of the present invention, is characterized in that comprising the following steps:
(1) initialization of synchro system: after service routine starts, according to configuration by system access data in advance graftabl frequently.And communicate with needing synchronous data center, compare the difference of original and copy, or with or without copy, then carry out synchronous.First according to same strategy, burst is carried out to original copy and counterpart relatively, the difference of more each burst, then carry out synchronously to differentiated burst;
(2) read and write access of data: the interface that the access of data is all provided by system directly internally deposit in buffer memory read and write, the service routine on backstage has one or more separate threads by the buffer memory write hard disk in internal memory.The service thread on backstage is taked periodically the content revised by interface batch to be write hard disk, and ensureing the persistence of data, is not that each amendment writes at once, neither periodically by all data write hard disks.
(3) remote synchronization of data: periodically the content revised by interface is directly synchronized to remote data center from internal memory is not that each amendment is synchronous at once, neither periodically synchronous all data.If the servers off-line of long-range reception, a large amount of internal memory amendments during this can be kept in local file by home server.When network recovery, file is sent to remote data center to upgrade, ensures the reliability of system.
(4) two-shipper of data server is standby mutually: standby host is built and after initialization completes, high-speed cache is also consistent with main frame.Now start the synchronized update receiving data, receive the content synchronization of amendment to internal memory and hard disk by network from main frame.When main frame machine back-up machine of delaying is taken over immediately and born service role and be namely upgraded to main frame, and continue to carry out synchronous with the server of remote data center.Can artificial selection continue serve as host role execution remote synchronization task or return to standby host state after the reparation of pending fault server.
Above-mentioned functions realizes by computer program, can operate in various UNIX and Linux, Windows operating system, and does not rely on the software that operating system carries.
The automatic switchover of 5 methods and main and standby competition mechanism, whether automatic decision carries out the switching of application state.Stream the present invention, by above step, can reach following technique effect:
1, synchronous efficiency is high, only carries out synchronously for the amendment part of data;
2, occupied bandwidth is little, adopts the burst manner of comparison of data, avoid initialization, normal synchronized or abnormal restoring time transmit all data;
3, reliability is high, if amendment can exist local by distance host off-line, treats that network recovery continues synchronous.And adopt two-shipper standby technology mutually;
4, portable good, all functions are have computer program to realize completely, may operate in various UNIX and Linux, Windows operating system, do not need the software carried by any operating system;
5, have cross-platform ability, source and destination can be deployed on the server of different operating system;
6, need the data of prestrain, the upper limit of high-speed cache, the Parameter Programmings such as the size of burst are adjustable;
7, dispose simply, only need dispose a service routine, dynamic base, a configuration file can run.
Accompanying drawing explanation
Fig. 1 is the synchronous flow chart of wide-area data between data center in the present invention;
Fig. 2 is reading and writing data browsing process figure in the present invention;
System initialization flow chart in Fig. 3 the present invention.
Embodiment
Below in conjunction with accompanying drawing, wide-area data synchronous method between the data center based on high-speed cache of the present invention is further described.
See Fig. 1 and Fig. 2, wide-area data synchronous method between a kind of data center based on high-speed cache of the present invention, its with remote data center synchronously for main purpose, improving local data access efficiency is secondary objective, with incremental update, increment synchronization and file backup for method, and it is high to provide a kind of local and remote access efficiency, the data volume of synchronous transmission is little, wide-area data synchronous method between the data center that also can ensure system conformance under the condition of remote server off-line a period of time, it comprises the following steps:
(1) synchronous object is all the real time data in internal memory, carries out synchronously first to read in internal memory to the file on disk;
(2) to the mode that the synchronous employing increment synchronization of internal storage data is combined with synchronize fully, data synchronous system provides data access interface to use to upper level applications, upper level applications is by the real time data in this interface amendment internal memory, the content synchronization that interface interchange is revised by synchro system is to remote data center, and remote data center performs same interface interchange and namely completes data syn-chronization;
(3), after the reception server off-line of remote data center, the internal memory amendment during this can be kept in local file and form backup by the server of local data center.When network recovery, this file is sent to teledata to upgrade, ensures the reliability of system;
(4) mode of the data acquisition incremental update in internal memory writes hard disk, ensures the persistence of data.The content write hard disk only will revised by interface, instead of periodically data are all write hard disk;
(5) remote data center adopts incremental update to the mode of hard disk, by the Update Table calling interface write hard disk received, instead of periodically by all data write hard disks;
(6) whether, during initialization data synchro system, data fragmentation is calculated MD5 hashed value by local and remote data center, consistent by the data of the comparison of computational results two places of burst.If inconsistent, immediately by long-range for being synchronized to of burst, solve inconsistence problems of other abnormal generations during system cloud gray model last time.
For more further setting forth the present invention, existing specific embodiment is as follows:
Fig. 1 is the synchronous flow chart of wide-area data between data center in the present invention.For the data set D of certain functional module
i, a series of retouching operation M is have received when synchronizing cycle does not arrive
ij (), operation is all stored in queue Ls (i) to be synchronized.The network state of now service routine inspection and remote data center, if interrupt, the direct content by queue Ls (i) is saved in file F
iin (k).If state is normal and arrive synchronizing cycle, queue content is sent to remote data center.Remote data center all receives, and sends it back synchronous successfully acknowledge message.If do not receive confirmation, the same content by queue Ls (i) is saved in file F
iin (k).Multiple synchronizing cycle, rear network was all abnormal in the past, can produce multiple backup file F
i.
If synchronous successful remote data center by calling interface by retouching operation M
ij () same purpose is in data set D ' (i).
Fig. 2 is reading and writing data browsing process figure in the present invention.
For the data set D of certain functional module
i, receive retouching operation M
ij (), upgrades high-speed cache at once.And when write back cycle does not arrive, these operations are all stored in be treated in write-back queue Lw (i).When write back cycle arrives, the revised context in queue Lw (i) is all write hard disk.
If read-only operation then can not operate write-back queue, directly read the data in high-speed cache.
System initialization flow chart in Fig. 3 the present invention.
The data set D of certain functional module during system initialization
ibe written into high-speed cache, and be divided into n sheet carry out MD5 value calculate.Then the MD5 value of n burst is transferred to remote data center.Wherein p, q, m block two places are inconsistent to suppose to receive rear discovery by remote data center, then numbering p, q, m are beamed back source data center.Local source data center subsequently by the transfer of data of p, q, m block to remote data center.Remote data center calling interface write cache.After each data block completes contrast, system initialization completes.
The present invention only carries out synchronously, improve synchronous efficiency for the amendment part of data; And occupied bandwidth is little, adopt the burst manner of comparison of data, avoid initialization, normal synchronized or abnormal restoring time transmit all data; If amendment can exist local by distance host off-line, treat that network recovery continues synchronous, add reliability.And adopt two-shipper standby technology mutually; It has portable good and cross-platform ability, and all functions are have computer program to realize completely, may operate in various UNIX and Linux, Windows operating system, do not need the software carried by any operating system; Source and destination can be deployed on the server of different operating system.
In addition, the present invention can the data of prestrain, the upper limit of high-speed cache, and the Parameter Programmings such as the size of burst are adjustable; And dispose simple, only need dispose a service routine, dynamic base, a configuration file can run.
In sum, the present invention is that a kind of synchronous efficiency is high, occupied bandwidth is little, reliability is high, portability is good, deployment is simple.
The present invention is illustrated according to the preferred embodiment, should be appreciated that but above-described embodiment does not limit the present invention in any form, the technical scheme that the form that all employings are equal to replacement or equivalent transformation obtains, and all drops within protection scope of the present invention.
Claims (4)
1. based on high-speed cache data center between a wide-area data synchronous method, it is characterized in that, it comprises the following steps:
(1) initialization of synchro system; The server of local data center according to configuring system access data graftabl frequently, and communicates with needing each synchronous remote data center, and then the difference of comparing data carry out synchronous;
(2) read and write access of data; Buffer memory in all directly internally being deposited by interface the access of data is read and write, and the service routine on backstage has one or more separate threads by the data write hard disk in internal memory;
(3) remote synchronization of data; Periodically the content that interface interchange is revised directly is synchronized to remote data center from internal memory, remote data center performs same interface interchange and namely completes data syn-chronization, the difference of described comparing data is that original compares with the data of copy, and is compared by MD5 hashed value; If the MD5 value of data and long-range inconsistent, be synchronized to copy from data original immediately;
In described step (2), the mode of what described data write disk adopted is incremental update, data synchronous system provides data access interface to use to upper level applications, and upper level applications is by the real time data in this interface amendment internal memory; The service routine on backstage has one or more separate threads periodically the content revised by interface batch to be write hard disk, for ensureing the persistence of data;
In described step (3), described remote data center adopts incremental update, to the mode of hard disk, the revised context calling interface received batch is write hard disk; The high-speed cache of local data center produces backup simultaneously in a standby host, when main frame machine back-up machine of delaying is taken over immediately and born service role, and continues to carry out synchronous with remote data center; After the reception server off-line of described remote data center, a large amount of internal memory amendments during this can be kept in local file and form backup by the server of local data center; When network recovery, this file is sent to teledata to upgrade.
2. wide-area data synchronous method between the data center based on high-speed cache according to claim 1, it is characterized in that: it is the burst original of data and copy being carried out certain particle size that described original compares with the data of copy, different bursts is produced MD5 value respectively and compares; Inconsistent burst after is relatively carried out synchronous to copy of original.
3. wide-area data synchronous method between the data center based on high-speed cache according to claim 1 and 2, is characterized in that: the high-speed cache of described remote data center produces the backup of data simultaneously in a standby host; When main frame machine back-up machine of delaying is taken over immediately and born service role, and continue the synchronous of the source server receiving original place.
4. wide-area data synchronous method between the data center based on high-speed cache according to claim 1, it is characterized in that: in described step (1), to the mode that the synchronous employing increment synchronization of internal storage data is combined with synchronize fully, data synchronous system provides data access interface to use to upper level applications, upper level applications is by the real time data in this interface amendment internal memory, the content synchronization that interface interchange is revised by synchro system is to remote data center, and remote data center performs same interface interchange and namely completes data syn-chronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210486206.0A CN103023879B (en) | 2012-11-26 | 2012-11-26 | Wide-area data synchronous method between a kind of data center based on high-speed cache |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210486206.0A CN103023879B (en) | 2012-11-26 | 2012-11-26 | Wide-area data synchronous method between a kind of data center based on high-speed cache |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103023879A CN103023879A (en) | 2013-04-03 |
| CN103023879B true CN103023879B (en) | 2015-11-18 |
Family
ID=47972010
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210486206.0A Active CN103023879B (en) | 2012-11-26 | 2012-11-26 | Wide-area data synchronous method between a kind of data center based on high-speed cache |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103023879B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103346904B (en) * | 2013-06-21 | 2016-03-30 | 西安交通大学 | A kind of fault-tolerant OpenFlow multi controller systems and control method thereof |
| CN103543958B (en) * | 2013-09-30 | 2017-02-15 | 华为技术有限公司 | Method and device for copying incremental data |
| CN104615594B (en) * | 2013-11-01 | 2018-04-03 | 阿里巴巴集团控股有限公司 | A kind of data-updating method and device |
| CN105721520B (en) * | 2014-12-02 | 2019-01-18 | 清华大学 | A kind of file synchronisation method and device |
| CN104572921B (en) * | 2014-12-27 | 2017-12-19 | 北京奇虎科技有限公司 | A kind of method of data synchronization and device across data center |
| CN105100264B (en) * | 2015-08-21 | 2019-06-07 | 福建天晴数码有限公司 | Optimize the method and system of bandwidth |
| CN106713393B (en) * | 2015-11-13 | 2020-06-02 | 阿里巴巴集团控股有限公司 | Data writing method and writing device |
| CN105391798B (en) * | 2015-12-04 | 2018-12-04 | 北京全路通信信号研究设计院集团有限公司 | A kind of active and standby control system method of data synchronization of train and device |
| CN105610957A (en) * | 2016-01-07 | 2016-05-25 | 中国联合网络通信集团有限公司 | Cloud system and data synchronization method thereof |
| CN105955983A (en) * | 2016-04-18 | 2016-09-21 | 国电南瑞科技股份有限公司 | Historical sampled data caching method for super-large scale power grid regulation and control system |
| CN106055281B (en) * | 2016-06-29 | 2020-02-18 | 广州华多网络科技有限公司 | Data writing method and device |
| CN106973089B (en) * | 2017-03-16 | 2020-07-17 | 北京索瑞特医学技术有限公司 | Data processing method and device |
| CN107483631B (en) * | 2017-09-19 | 2020-04-07 | 山东大学 | Method for controlling cache to realize mobile internet service access |
| CN108712505B (en) * | 2018-05-31 | 2021-04-06 | 北京百度网讯科技有限公司 | Data synchronization method, device, equipment, system and storage medium |
| CN109672717B (en) * | 2018-09-25 | 2023-02-03 | 平安普惠企业管理有限公司 | Method, device, storage medium and apparatus for maintaining system consistency |
| CN112416656B (en) * | 2020-12-08 | 2022-08-26 | 卡斯柯信号有限公司 | Data offline storage method for station application server |
| CN112948336B (en) * | 2021-03-30 | 2023-01-03 | 联想凌拓科技有限公司 | Data acceleration method, cache unit, electronic device and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833486A (en) * | 2010-04-07 | 2010-09-15 | 山东高效能服务器和存储研究院 | Method for designing remote backup and recovery system |
| CN102624871A (en) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | Remote file synchronizing method realized based on distributed enterprise service bus |
-
2012
- 2012-11-26 CN CN201210486206.0A patent/CN103023879B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833486A (en) * | 2010-04-07 | 2010-09-15 | 山东高效能服务器和存储研究院 | Method for designing remote backup and recovery system |
| CN102624871A (en) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | Remote file synchronizing method realized based on distributed enterprise service bus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103023879A (en) | 2013-04-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103023879B (en) | Wide-area data synchronous method between a kind of data center based on high-speed cache | |
| US10778751B2 (en) | Method of micro-service transformation for power trading functions | |
| CN102521038B (en) | Virtual machine migration method and device based on distributed file system | |
| US10152527B1 (en) | Increment resynchronization in hash-based replication | |
| CN108287835B (en) | Data cleaning method and device | |
| CN106469085B (en) | The online migration method, apparatus and system of virtual machine | |
| EP3195574B1 (en) | Techniques for coordinating parallel performance and cancellation of commands in a storage cluster system | |
| CN100371851C (en) | Time synchronizing method and device | |
| CN104202440B (en) | A kind of method of marking terminal, server and system | |
| CN105138615A (en) | Method and system for building big data distributed log | |
| US20100049932A1 (en) | Method and apparatus for automatic snapshot | |
| CN104468521B (en) | Online moving method, device and system | |
| CN105740248B (en) | A kind of method of data synchronization, apparatus and system | |
| CN102981931A (en) | Backup method and device for virtual machine | |
| CN103970585A (en) | Method and device for creating virtual machine | |
| CN104461654A (en) | OBD equipment function unit and firmware classification optional upgrading method | |
| CN104714811A (en) | Method and device for manufacturing difference upgrade package and system difference upgrade method and device | |
| CN111124277A (en) | Deep learning data set caching method, system, terminal and storage medium | |
| CN106354865B (en) | Method, device and system for synchronizing master database and slave database | |
| CN113987064A (en) | Data processing method, system and equipment | |
| CN111651275A (en) | MySQL cluster automatic deployment system and method | |
| CN104021200A (en) | Data synchronizing method and device of database | |
| CN104461706B (en) | A kind of method and multiprocessing device that shared global variable is shared | |
| CN102508886A (en) | Extensive makeup language (XML)-based method for synchronously updating increment of spatial data | |
| CN104424275A (en) | Database system and data synchronization method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Gao Yuan Inventor after: Hu Xunhui Inventor after: Xu Hongqiang Inventor after: Song Xuri Inventor after: Chen Peng Inventor after: Wang Dieyu Inventor after: Zhou Ji Inventor after: Ge Chaoqiang Inventor after: Zhang Xiaobai Inventor after: Chang Naichao Inventor after: Wang Lei Inventor after: Gu Wenjie Inventor after: Ren Sheng Inventor after: Sha Yichuan Inventor after: Zhou Jieying Inventor after: Chen Guoping Inventor after: Wang Xingzhi Inventor after: Liu Jinbo Inventor after: Zhao Li Inventor before: Gao Yuan Inventor before: Gu Wenjie Inventor before: Ren Sheng Inventor before: Sha Yichuan |