CN112699122A - Method and system for synchronous processing of bit number configuration data under distributed condition - Google Patents
Method and system for synchronous processing of bit number configuration data under distributed condition Download PDFInfo
- Publication number
- CN112699122A CN112699122A CN202011589872.8A CN202011589872A CN112699122A CN 112699122 A CN112699122 A CN 112699122A CN 202011589872 A CN202011589872 A CN 202011589872A CN 112699122 A CN112699122 A CN 112699122A
- Authority
- CN
- China
- Prior art keywords
- data
- version number
- bit number
- change
- configuration data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 16
- 230000008859 change Effects 0.000 claims abstract description 72
- 238000003672 processing method Methods 0.000 claims abstract description 12
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 abstract description 6
- 238000013070 change management Methods 0.000 abstract description 2
- 238000012986 modification Methods 0.000 description 12
- 230000004048 modification Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for synchronously processing bit number configuration data under distributed conditions, wherein the data structure of the bit number configuration data at least comprises a unique bit number index, a stable creation version number determined based on the bit number index, a change version number changed based on system data change and a update version number changed based on data change of the bit number; the data synchronization processing method specifically comprises the steps of obtaining a global operation version number; carrying out archiving backup on the original data; updating and storing data based on the bit number index, and updating a change version number in a bit number configuration data structure into a global operation version number during storage; a bit number configuration data synchronous processing system is also provided; the method realizes a synchronous management mode of the distributed multi-version bit number data configuration, realizes rapid configuration synchronization and version change management aiming at large-scale data bit numbers in the distributed field, and ensures the consistency of the bit number data configuration among multi-node distributed systems.
Description
Technical Field
The invention relates to the field of industrial control, in particular to a method and a system for synchronous processing of bit number configuration data in a distributed situation.
Background
In the industrial control field, due to the proposal of a large-scale system and a distributed architecture, the usage amount of the field point location configuration data which is the most important ring in field data acquisition is increased along with the gradual increase of the scale of the system, and the small-scale hundreds of orders of magnitude of the small-scale system is gradually increased to tens of thousands or even hundreds of thousands of orders of magnitude. If the data analysis processing center is a more upper-level large data analysis field or a summary analysis field, such as a plurality of plants or the data analysis processing center of the whole industry, the order of magnitude is very large.
The synchronization processing of the point location configuration data, which directly affects the field production, the index accuracy and the data reuse rate, becomes an important part, and at present, the synchronization of the point location configuration data is divided into incremental synchronization and full synchronization, which have certain disadvantages and limitations. The full synchronization is performed by reading all information, and the synchronized end is required to compare the information of the synchronized end with the full information one by one and then change the information, so that the synchronized end has very excellent performance and synchronization results when the magnitude is low, but the performance deviation will rise exponentially as the magnitude is enlarged, and the load of the whole synchronized system will rise greatly as the complexity of comparison data increases.
Incremental synchronization, that is, a changed party is informed of the corresponding change situation by means of active notification from a changing party, and since the changed party needs to be informed actively by the changing party, there is a certain requirement constraint on the structure of the whole system, and meanwhile, because of the instability of the distributed system, that is, the communication goodness between system nodes cannot be guaranteed by 100%, the informing party needs more and more complicated exception handling and state handling as more nodes need to be informed, and meanwhile, if the changing party is a multi-node, the complexity of the system rises linearly, and the steady state thereof is difficult to guarantee.
Therefore, the synchronization of a large amount of data brings about many unsolved troubles for the responsiveness, maintainability and data comparability of the system.
Disclosure of Invention
Aiming at the problems, the invention provides a method for synchronously processing the bit number configuration data under the distributed condition, which solves the problem of effectively and consistently synchronizing a large number of data configurations among a plurality of different nodes and systems, improves the synchronization efficiency, reduces unnecessary data interaction and comparison judgment processing logic to avoid the loss of the system performance, and further improves the throughput rate and the execution efficiency of the system.
The second objective of the present invention is to provide a system for synchronously processing bit number configuration data under distributed conditions.
In order to achieve the above object, the data structure of the bit number configuration data at least includes a unique bit number index, a stable creation version number determined based on the bit number index, a change version number changed based on a change of system data, and an update version number changed based on a change of data of the bit number itself; the data synchronization processing method comprises the following steps: acquiring a global operation version number corresponding to system data change, and synchronizing data based on the global operation version number and a bit number configuration data structure; carrying out archiving backup on the original data; and updating and storing the data based on the bit number index, and updating the change version number in the bit number configuration data structure into a global operation version number based on the global operation version number during storage.
Further, the method also comprises the step of carrying out consistency processing on the update version number and the global operation version number; and after the consistency processing, discarding and/or merging operation log data before the global operation version number based on the global operation version number.
Further, the update version number changes based on changes in its own data, or a biased update version number is calculated based on the global operation version number.
Furthermore, the created version number, the changed version number and the updated version number are unsigned linear 64-bit integers and are changed in a linear increasing mode when changed.
Further, the global operation version number is sequentially linearly incremented based on any data changes within the system.
Further, the version number is created for processing and judging the new configuration data of the bit number, including but not limited to the deleted new configuration data of the bit number and the completely new configuration data of the bit number.
Further, the method also comprises the step of retrieving the operation record change of the bit number based on the range between the creation version number and the change version number.
Further, the operation record change of the bit number includes, but is not limited to, deletion, addition after deletion, and complete addition.
In order to achieve the second objective of the present invention, the present invention provides a system for synchronously processing bit number configuration data under distributed conditions, which at least comprises a central node and a service node; the central node generates a corresponding linearly-increased global operation version number based on data change in the system, and actively sends the global operation version number to the service node based on a request of the service node or the initiative; and the service node synchronizes the bit number configuration data based on the global operation version number and updates the change version number of the bit number configuration data during data storage.
Furthermore, the service node updates the update version number of the bit number configuration data based on the data change of the service node.
Further, the service node coordinates and processes data synchronization based on the update version number and the global operation version number.
The invention has the following beneficial technical effects:
1. the invention redefines the data structure of the bit number configuration data aiming at the bit number configuration data synchronization under the distributed condition, realizes multi-version management and configuration data synchronization on the basis of the redefined data structure, and ensures the consistency of the bit number data configuration among multi-node distributed systems.
2. Based on the linear increment of the changed version number, the created version number and the updated version number when the change occurs, the bit number configuration data has the capability of presenting multi-dimensional indexes, thereby realizing the quick retrieval of different dimensions.
3. The multi-index design mode can be expanded and realized on the basis of the search mode for specific function application to facilitate the development of upper-layer functions, and realize tracking any change based on the change or confirming all result snapshots up to which the change is made after a certain change based on the result.
4. The implementation scheme of universality can meet the requirements of other data synchronization by the expansion, and balance between performance and snapshot is realized.
5. Data synchronization errors due to time section synchronization or time differences between different nodes are reduced.
Drawings
FIG. 1 is a schematic diagram of a multi-node distributed system according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of a bit number configuration data structure according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating a bit number configuration data synchronization processing method according to an embodiment of the present invention.
Fig. 4 is a schematic diagram illustrating logic changes of global operation version numbers in an embodiment of a bit number configuration data synchronization processing method according to the present invention.
Fig. 5 is a schematic diagram illustrating the determination of the number change based on the created version number and the changed version number in the embodiment of the synchronous processing method for the number configuration data of the present invention.
Fig. 6 is a schematic diagram illustrating a data synchronization and self data change flow in the bit number configuration data synchronization processing method according to the present invention.
FIG. 7 is a diagram illustrating an embodiment of a system for synchronous processing of bit number configuration data according to the present invention.
Detailed Description
For a further understanding of the invention, reference will now be made to the preferred embodiments of the invention by way of example, and it is to be understood that the description is intended to further illustrate features and advantages of the invention, and not to limit the scope of the claims.
The invention provides a bit number configuration data synchronous processing method applied to the distributed condition,
the invention aims to solve the problem of how to perform rapid configuration synchronization and version change management on large-scale data bit numbers in the distributed field, designs a set of bit number configuration data synchronization processing method and system applied to the distributed situation, and the detailed description is provided below by combining with the embodiment.
For a more detailed and clear explanation of the bit number configuration data synchronization processing method proposed in the present invention, reference is made to the embodiment of the multi-node distributed system shown in fig. 1.
In the following illustrative embodiments of the present invention, the bit number configuration data is information set data composed of a series of parameters such as a bit number name, a bit number description, a type, an upper limit value, a lower limit value, and the like, and the bit number configuration data is used as a basis for the upper application system to perform the acquisition of field data and the processing of a calculation process.
The multi-node distributed system shown in FIG. 1 includes a unified management processing unit and A, B, C nodes, wherein the A node includes a data bit number A, B, the B node includes a data bit number C, D, and the C node includes a data bit number E, F.
The bit number point data has a globally unique ID in the system and presents an obvious distributed grouping structure, so that the data structure shown in fig. 2 is provided by taking the bit number point data as a main data index, and multi-version information recording and management are realized to ensure that the variation of the whole system is changed linearly.
In an embodiment of the present invention, the data structure of the defined bit number configuration data is as shown in fig. 2, and at least includes a unique bit number index, a stable creation version number determined based on the bit number index, a change version number changed based on a change of system data, and an update version number changed based on a change of data of the bit number itself; the data synchronization processing method is shown in fig. 3, and includes: s1, acquiring a global operation version number corresponding to the change of the system data, and synchronizing the data based on the global operation version number and a bit number configuration data structure; s2, archiving and backing up the original data; and S3, updating and storing the data based on the bit number index, and updating the change version number in the bit number configuration data structure into a global operation version number based on the global operation version number during storage.
In some embodiments, when the bit number data is created, its corresponding creation Version number Create Version will be determined and its value will not change. The modification Version number modification Version and the Update Version number Update Version are respectively given initial values when being created, and the values are correspondingly changed according to global data change and self data change.
For a node, the change Version number modification Version can correspond to a global Operation Version number Operation Version of a system, is definite data, namely an actual Operation log, is linear and unique on the system, and can be logically processed on the basis of time.
For the node, the Update Version number Update Version is mainly used for coordinating external Version management of the node, when the Update Version number Update Version changes, the Update Version number Update Version is correspondingly modified, and the modification and the corresponding change do not affect other nodes.
The three version numbers of the configuration data structure of the bit number are 64-bit unsigned integers, the change follows the increasing operation, even if the operation frequency is changed once for 1ms, the change can be used for about 584942417 years and far exceeds the upper limit of the whole system, therefore, the relevant overflow condition can not be generated, and the linear process of the change of the configuration data structure of the bit number is ensured.
It is the global Operation Version number Operation Version that needs to be synchronized for all nodes of the system. Any data change of any node can make the global Operation Version number Operation Version of the system realize linear increment according to a certain rule and sequence to obtain corresponding, such as permanent increment processing of the global Operation Version number Operation Version shown in fig. 4.
In some embodiments, for the data node a, the bit number index a has uniqueness, when the data corresponding to the bit number a in the system is updated, the generated global Operation Version number Operation Version is 10087, the data node a performs archive backup on the original data before the change according to the bit number index, and due to time synchronization or a fixed time difference between different nodes, the Update Version number Update Version and the global Operation Version number Operation Version perform corresponding consistency processing, for example, after the Update Version number Update Version calculates a deviation through the global Operation Version, calibration is performed based on the difference. After the consistency processing, merging processing is performed once according to the Operation Version, that is, Operation log data before the global Operation Version number Operation Version is discarded and merged. When the data is stored, the Version number modification Version is changed, namely, the Version number modification Version is updated to the value of the global Operation Version number Operation Version.
In some embodiments, when data of a certain number of a single node changes, linear operation consistency is obtained through local timestamp Update Version, and when the data is submitted to the whole central node to synchronize the whole system node, global linear consistency is obtained through obtaining corresponding global Modify Version.
As shown in fig. 5, when the bit number a configuration data changes once, the corresponding Update Version number Update Version becomes 1, and is submitted to the central node to synchronize the entire system node, the global Operation Version number Operation Version of the central node is linearly incremented to 10087, and the change Version number modification Version of the corresponding bit number a configuration data structure is updated to 10087. The bit number a configuration data is changed again, the corresponding Update Version number Update Version is linearly increased to 2 and submitted to the central node to synchronize the whole system node, the global Operation Version number Operation Version of the central node is linearly increased to 10089, and the system also generates other data changes, so the updated data of the change Version number Modify Version of the bit number a configuration data structure is 10089.
Therefore, the single node can obtain the change condition of the corresponding bit number and obtain the corresponding change result according to the change Version number Modify Version. The single node only needs to obtain the difference change between the modification Version number Modify Version and the central node when the single node wants to know the change of other nodes. From the perspective of the whole system, whether a change difference of a limited range or a global change difference is obtained can be handled according to a linear relationship between the change Version number modification Version and the Update Version number Update Version.
It should be noted that, in the aspect of performing change monitoring or obtaining a change result, all changes after the whole system may be obtained according to the change Version number modification Version determined by the node, or all final bit numbers of a certain determined node may be obtained according to the Update Version number Update Version, so as to implement multi-Version request management, and quickly index and locate data required by itself according to actual needs.
In some embodiments of the present invention, all change operations to the bit number a may also be retrieved from the creation Version number Create Version and the change Version number Modify Version. Because the creation Version number Create Version may be newly added and generated completely or after deletion. The creation Version number Create Version is a certain point of the change Version number Modify Version. The query formula shown in fig. 6 can be used to trace back the operation records including the deletion.
The invention also provides a bit number configuration data synchronization system, which at least comprises a central node and a service node as shown in fig. 7, in another embodiment of the invention, the central node generates a corresponding linearly-increased global operation version number based on data change in the system, and sends the global operation version number to the service node based on a request of the service node or actively, after the service node is consistent and processed based on the updated version number and the global operation version number, the bit number configuration data is synchronized, and the change version number of the bit number configuration data is updated during data storage.
The above description of the embodiments is only intended to facilitate the understanding of the method of the invention and its core idea. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (11)
1. A method for synchronously processing bit number configuration data under distributed conditions is characterized in that a data structure of the bit number configuration data at least comprises a unique bit number index, a stable creation version number determined based on the bit number index, a change version number changed based on system data change and an update version number changed based on data change of the bit number; the data synchronization processing method comprises the following steps:
acquiring a global operation version number corresponding to system data change, and synchronizing data based on the global operation version number and a bit number configuration data structure;
carrying out archiving backup on the original data;
and updating and storing the data based on the bit number index, and updating the change version number in the bit number configuration data structure into a global operation version number based on the global operation version number during storage.
2. The method of claim 1, further comprising the step of synchronizing the number of bits configuration data,
the updating version number is consistent with the global operation version number;
and after the consistency processing, discarding and/or merging operation log data before the global operation version number based on the global operation version number.
3. The method according to claim 2, wherein the update version number is changed based on a change in its own data, or a biased update version number is calculated based on a global operation version number.
4. The method of claim 1, wherein the creation version number, the change version number, and the update version number are unsigned linear 64-bit integers that change incrementally and linearly when changed.
5. The method of claim 1, wherein the global operation version number is sequentially and linearly incremented based on any data change in the system.
6. The method as claimed in claim 1, wherein the created version number is used to process and determine new configuration data, including but not limited to, the new configuration data after deletion and the completely new configuration data.
7. The method for synchronously processing bit number configuration data according to any one of claims 1 to 6, further comprising retrieving an operation record change of the bit number based on a range between the creation version number and the change version number.
8. The method as claimed in claim 7, wherein the operation record change of the bit number includes but is not limited to delete, add after delete and completely add.
9. A synchronous processing system of configuration data of bit number applying the synchronous processing method of configuration data of bit number of any claim 1-8, characterized in that, at least includes a central node and a service node; the central node generates a corresponding linearly-increased global operation version number based on data change in the system, and actively sends the global operation version number to the service node based on a request of the service node or the initiative;
and the service node synchronizes the bit number configuration data based on the global operation version number and updates the change version number of the bit number configuration data during data storage.
10. The system of claim 9, wherein the service node updates the updated version number of the bit number configuration data based on the change of its own data.
11. The system for processing synchronization of bit number configuration data of claim 10, wherein the service node coordinates processing of data synchronization based on an update version number and a global operation version number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011589872.8A CN112699122A (en) | 2020-12-29 | 2020-12-29 | Method and system for synchronous processing of bit number configuration data under distributed condition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011589872.8A CN112699122A (en) | 2020-12-29 | 2020-12-29 | Method and system for synchronous processing of bit number configuration data under distributed condition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112699122A true CN112699122A (en) | 2021-04-23 |
Family
ID=75513079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011589872.8A Pending CN112699122A (en) | 2020-12-29 | 2020-12-29 | Method and system for synchronous processing of bit number configuration data under distributed condition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112699122A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006340342A (en) * | 2005-06-03 | 2006-12-14 | Lg Electronics Inc | Method for managing digital multimedia broadcasting channel, digital multimedia broadcasting system, and terminal capable of receiving digital multimedia broadcast |
| CN108259562A (en) * | 2017-12-11 | 2018-07-06 | 杭州品茗安控信息技术股份有限公司 | A kind of method of data synchronization and device based on multi-endpoint |
| CN109299194A (en) * | 2018-09-25 | 2019-02-01 | 平安科技（深圳）有限公司 | Multi-edition data memory management method and device, electronic equipment, storage medium |
| CN111752488A (en) * | 2020-06-30 | 2020-10-09 | 重庆紫光华山智安科技有限公司 | Management method and device of storage cluster, management node and storage medium |
-
2020
- 2020-12-29 CN CN202011589872.8A patent/CN112699122A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006340342A (en) * | 2005-06-03 | 2006-12-14 | Lg Electronics Inc | Method for managing digital multimedia broadcasting channel, digital multimedia broadcasting system, and terminal capable of receiving digital multimedia broadcast |
| CN108259562A (en) * | 2017-12-11 | 2018-07-06 | 杭州品茗安控信息技术股份有限公司 | A kind of method of data synchronization and device based on multi-endpoint |
| CN109299194A (en) * | 2018-09-25 | 2019-02-01 | 平安科技（深圳）有限公司 | Multi-edition data memory management method and device, electronic equipment, storage medium |
| CN111752488A (en) * | 2020-06-30 | 2020-10-09 | 重庆紫光华山智安科技有限公司 | Management method and device of storage cluster, management node and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10754875B2 (en) | Copying data changes to a target database | |
| EP3602341B1 (en) | Data replication system | |
| US8656270B2 (en) | Change-oriented spreadsheet application | |
| US9124612B2 (en) | Multi-site clustering | |
| US10783122B2 (en) | Method and apparatus for recording and managing data object relationship data | |
| US20150339308A1 (en) | Managing site-based search configuration data | |
| US20080307011A1 (en) | Failure recovery and error correction techniques for data loading in information warehouses | |
| WO2016118964A1 (en) | Application-centric object storage | |
| CN107391635A (en) | Data synchronous system and method | |
| CN111506556A (en) | Multi-source heterogeneous structured data synchronization method | |
| CN109299121B (en) | Efficient historical alarm query method for SCADA system | |
| CN111708895B (en) | Knowledge graph system construction method and device | |
| Ma et al. | Log-based change data capture from schema-free document stores using mapreduce | |
| CN114153809A (en) | Parallel real-time incremental statistic method based on database logs | |
| CN116501700B (en) | APP formatted file offline storage method, device, equipment and storage medium | |
| CN114595286A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN108491218A (en) | A kind of method and apparatus of databases comparison file generated | |
| CN112699122A (en) | Method and system for synchronous processing of bit number configuration data under distributed condition | |
| CN111858767A (en) | Synchronous data processing method, device, equipment and storage medium | |
| CN112162992A (en) | Efficient database updating system and method | |
| EP4336375A1 (en) | Data processing method and apparatus | |
| US10614095B2 (en) | Method and systems for monitoring changes for a server system | |
| CN115658815A (en) | CDC (control data center) -based data synchronization method | |
| CN115470192A (en) | Method for collecting and managing metadata of database | |
| CN111538713B (en) | Hive-oriented multi-mode data processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210423 |