CN106339262A - Data synchronization method in parallel calculation - Google Patents
Data synchronization method in parallel calculation Download PDFInfo
- Publication number
- CN106339262A CN106339262A CN201610756555.8A CN201610756555A CN106339262A CN 106339262 A CN106339262 A CN 106339262A CN 201610756555 A CN201610756555 A CN 201610756555A CN 106339262 A CN106339262 A CN 106339262A
- Authority
- CN
- China
- Prior art keywords
- information
- data
- computing unit
- data synchronization
- calculation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The invention disclsoes a data synchronization method in parallel calculation. The data synchronization method comprises the following steps: (1) after changing a data state by calculation units, issuing change information to a Topic channel of a message queue; (2) transmitting information by each calculation unit through the Topic channel; (3) after receiving the information by the other calculation units, changing and synchronizing the data state carried by the information to local cache. The data synchronization method in the parallel calculation has the advantages that after data is locally modified, a message is issued to a message system and the message system can synchronize to the other calculation units in real time; the data is issued and changed to an asynchronous mode, which is equivalent to broadcasting to each calculation unit without concerning how many other calculation units need to be synchronized, so that performances of a local system are not influenced no matter how many parallel calculation units are arranged; the operation of previous calculation units are not affected while the operation units are increased, and a native machine only needs to be registered to the message system in a deployment process.
Description
Technical field
The present invention relates to areas of information technology, the method for data synchronization in specially a kind of parallel computation.
Background technology
In used car value appraisal system, need to process more than one hundred million data at short notice, for speed up processing, adopt
With multiple stage computers method for parallel processing.
In parallel process, the synchronization to data source data state becomes one of key it is ensured that same data source will not
It is repeatedly processed through vaporization, condensation and absorption or fault processing it is therefore desirable to be synchronized to multiple computing units in time to guarantee each calculating by data mode
Unit can promptly and accurately obtain data mode.
Under normal circumstances, multiple stage computing unit can pass through to access same data base, thus avoiding data synchronization problems,
Or access disparate databases, then synchronous by realizing between data base.For the former, when concurrency is larger, can be to data base
Cause very big pressure, thus causing database performance can become the bottleneck in parallel computation.For the latter, same between data base
Step scheme enforcement cost performance for general data syn-chronization is not high, and is difficult to Quick Extended parallel computation quantity.
Content of the invention
The present invention, in order to solve above-mentioned technical problem, provides the method for data synchronization in a kind of parallel computation.The mesh realized
For meet after a computing unit updates certain data mode can real-time synchronization give every other computing unit.
In order to reach above-mentioned technique effect, the present invention includes a kind of technical scheme below: parallel computation of present invention design
In method of data synchronization, comprise the steps, (1) computing unit change data mode after, change information is published to message
The topic passage of queue；
(2) each computing unit all enters the transmission of row information by topic passage；
(3), after other computing units receive information, the data mode change entrained by this information is synchronized to and locally delays
Deposit.
Further, the data mode after the described change information of step (1) includes changing, the publisher of this information.
Further, after in step (3), computing unit receives information, single with this calculating to publisher's computing unit information
Metamessage carries out judgement and compares, and publisher's computing unit information is consistent with this computing unit information, then make and ignore process, if not
Unanimously, then make synchrodata Status Change to local cache.
Using technique scheme, including following beneficial effect: 1 is quasi real time synchronous: after locally modified data sends out message
To after message system, message system can quasi real time be synchronized to other computing units to cloth；
2nd, do not affect local system performance: issuing this data change is asynchronous system, be equivalent to and be broadcast to each calculating list
Unit, but do not need to pay close attention to that how many other computing unit needs to be synchronized, and therefore parallel computation element number is how many all
Do not affect local system performance；
3rd, deployment is convenient and swift: increasing computing unit does not affect the operation of original computing unit, only needs to this during deployment
Machine is registered to message system.
Brief description
Fig. 1 the inventive method flow chart；
Specific embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing of the present invention, to the present invention
Technical scheme in embodiment is clearly and completely described it is clear that described embodiment is present invention part enforcement
Example, rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creativeness
The every other embodiment being obtained under the premise of work, broadly falls into the scope of protection of the invention.
Below by specific embodiment and combine accompanying drawing the present invention is described in further detail.Shown in reference Fig. 1,
Method of data synchronization in the parallel computation that the present invention provides, in processing data, especially increasingly tends to big data now
In the epoch, more than one hundred million data processings is got up, and in order to improve arithmetic speed, ensure accuracy, generally requires by multiple parallel meters
Calculate cell processing data, after one of computing unit change data mode, this issue computing unit will be changed information and issue
To the topic passage of message queue, each computing unit all enters the transmission of row information by topic passage, and change information includes
Data mode after change, the publisher of this information, so, when other computing units receive change information, due to message system
System is all reciprocity for all computing units, and therefore message issuer also can receive this message, but comprises due in message
Distributor information, the such as mark of publisher's computing unit, serial number, mac address etc., thus other computing units can according to send out
Cloth person's information carries out judgement with this computing unit and compares, and such as after comparing, is judged as that this computing unit is exactly publisher, then
Make and ignore process, if after comparing, being judged as it not being publisher, then make synchrodata Status Change to local cache
Process.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.All within the spirit and principles in the present invention, made any repair
Change, equivalent, improvement etc., should be included within the scope of the present invention.
Claims (4)
1. the method for data synchronization in a kind of parallel computation is it is characterised in that the method comprises the steps, (1) computing unit
After change data mode, change information is published to the topic passage of message queue；
(2) each computing unit all enters the transmission of row information by topic passage；
(3), after other computing units receive information, the data mode change entrained by this information is synchronized to local cache.
2. according to claim 1 the method for data synchronization in parallel computation it is characterised in that more converting to described in step (1)
Breath includes the data mode after change, the publisher of this information.
3. according to claim 2 the method for data synchronization in parallel computation it is characterised in that computing unit in step (3)
After receiving information, with this computing unit information, judgement is carried out to publisher's computing unit information and compares, publisher's computing unit
Information is consistent with this computing unit information, then make and ignore process, if inconsistent, makes synchrodata Status Change to locally
The process of caching.
4. according to claim 2 the method for data synchronization in parallel computation it is characterised in that described computing unit publisher
Information includes mark, serial number, mac address.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610756555.8A CN106339262A (en) | 2016-08-26 | 2016-08-26 | Data synchronization method in parallel calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610756555.8A CN106339262A (en) | 2016-08-26 | 2016-08-26 | Data synchronization method in parallel calculation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106339262A true CN106339262A (en) | 2017-01-18 |
Family
ID=57823392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610756555.8A Pending CN106339262A (en) | 2016-08-26 | 2016-08-26 | Data synchronization method in parallel calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106339262A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061654A (en) * | 2019-11-11 | 2020-04-24 | 支付宝(杭州)信息技术有限公司 | Cache refreshing processing method and device and electronic equipment |
| CN111400407A (en) * | 2020-04-10 | 2020-07-10 | 浙江大华技术股份有限公司 | Data synchronization method and device, storage medium and electronic device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
-
2016
- 2016-08-26 CN CN201610756555.8A patent/CN106339262A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061654A (en) * | 2019-11-11 | 2020-04-24 | 支付宝(杭州)信息技术有限公司 | Cache refreshing processing method and device and electronic equipment |
| CN111400407A (en) * | 2020-04-10 | 2020-07-10 | 浙江大华技术股份有限公司 | Data synchronization method and device, storage medium and electronic device |
| CN111400407B (en) * | 2020-04-10 | 2023-09-26 | 浙江大华技术股份有限公司 | Data synchronization method and device, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103699580B (en) | Database synchronization method and device | |
| CN109815373B (en) | Data storage control method and device, server and readable storage medium | |
| CN107193643B (en) | A kind of highly reliable distributed traffic real-time statistical method and system | |
| CN106339274A (en) | Method and system for obtaining data snapshot | |
| CN109614262A (en) | Business method of calibration, device and computer readable storage medium | |
| CN106850258A (en) | A kind of Log Administration System, method and device | |
| CN105354290A (en) | Method and system for searching specific personnel based on MAC address of mobile terminal | |
| CN106528871A (en) | Method for updating information of engineering data points in industrial control system on line | |
| CN101317375A (en) | Network management data synchronous refreshing method, and client terminal and server terminal | |
| CN107273496A (en) | A kind of detection method of micro blog network region accident | |
| CN106339262A (en) | Data synchronization method in parallel calculation | |
| CN108399231A (en) | A kind of collecting method and Flume data collection clients | |
| CN105843933A (en) | Index building method for distributed memory columnar database | |
| CN110175152A (en) | A kind of log inquiring method, transfer server cluster and log query system | |
| CN104636395A (en) | Count processing method and device | |
| CN105205189A (en) | BIM based on container and integrated method of high-speed data collecting system | |
| CN112347143A (en) | Multi-data stream processing method, device, terminal and storage medium | |
| CN115017159A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN112231285A (en) | Knowledge graph generation method and device based on data resources | |
| CN106547584A (en) | A kind of backing method and device of application program | |
| CN114297292A (en) | Data synchronization system based on canal platform and execution method | |
| CN104933119A (en) | Big data management method | |
| CN110472972A (en) | A kind of block chain storing data synchronisation snapshot method and system | |
| CN105205168A (en) | Exposure system based on Redis database and operation method thereof | |
| CN114238834A (en) | Real-time processing method, device and system based on publishing and subscribing Internet of things big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 210000 Gulou District, Nanjing, Jiangsu Province, 518, Block C, No. 4, Gupinggang, Nanjing Applicant after: Nanjing Sanbaiyun Information Technology Co., Ltd. Address before: 210000 Gulou District, Nanjing, Jiangsu Province, 518, Block C, No. 4, Gupinggang, Nanjing Applicant before: Nanjing car easy to Amoy network information technology Co., Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170118 |
|
| RJ01 | Rejection of invention patent application after publication |