CN112422616A - Distributed data transmission control system and method - Google Patents
Distributed data transmission control system and method Download PDFInfo
- Publication number
- CN112422616A CN112422616A CN202010984787.5A CN202010984787A CN112422616A CN 112422616 A CN112422616 A CN 112422616A CN 202010984787 A CN202010984787 A CN 202010984787A CN 112422616 A CN112422616 A CN 112422616A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- task
- item
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000013524 data verification Methods 0.000 claims abstract description 22
- 238000007726 management method Methods 0.000 claims abstract description 16
- 238000009826 distribution Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000012795 verification Methods 0.000 claims description 14
- 238000012544 monitoring process Methods 0.000 claims description 12
- 238000004458 analytical method Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 7
- 230000006978 adaptation Effects 0.000 claims description 6
- 230000001360 synchronised effect Effects 0.000 claims description 6
- 238000012790 confirmation Methods 0.000 claims description 4
- 230000003111 delayed effect Effects 0.000 claims description 3
- 230000001419 dependent effect Effects 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 238000004088 simulation Methods 0.000 claims description 3
- 239000013589 supplement Substances 0.000 claims description 2
- 238000013479 data entry Methods 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
- H04L1/0061—Error detection codes
Abstract
The invention relates to the technical field of data transmission, and particularly discloses a distributed data transmission control system and a distributed data transmission control method. The system comprises a data checking module, a task scheduling module and a data synchronization module, wherein the data checking module is used for checking data of task data which are connected and transmitted to an EAM; the task scheduling module and the data synchronization module are connected with the data verification module, and when data passes through data verification, the task scheduling module is used for performing sequence management and distribution scheduling on data synchronization tasks transmitted to the EAM, and the data synchronization module is used for performing synchronization processing on actual data transmitted to the EAM. The system and the method adopt the data transmission control of a mechanism of firstly checking and then synchronizing, and after checking, users can perform steady data synchronization work in a mode of simulating actual operation only by data synchronization, thereby remarkably liberating the workload of the users.
Description
Technical Field
The invention belongs to the technical field of data transmission, and particularly relates to a distributed data transmission control system and method.
Background
An enterprise asset management system (EAM for short) mainly solves the process and business management standardization in the nuclear power production management field, and is an important management tool for production standard business processes. EAM is one of the indispensable important links in power plant production management no matter in the transfer, delivery and production stages.
Compared with other foreign information systems which need the EAM data transmission function, the system has the following research findings: the transmission mode is mainly data synchronization, and detailed data detection and analysis are not performed on data before a synchronization stage; the calling mode is coupled inside the system in the form of an internal interface, so that the complexity of the system is improved, and services are not provided for other systems; the task processing mechanism mostly adopts a timing task polling mode, and cannot realize real-time task processing. At present, a unified platform for providing data transmission service for systems in enterprises is not provided for EAM domestically.
Disclosure of Invention
The invention aims to provide a distributed data transmission control system and a distributed data transmission control method, which solve the problem of data transmission of an enterprise asset management system and an enterprise internal system.
The technical scheme of the invention is as follows: a distributed data transmission control system comprises a data verification module, a task scheduling module and a data synchronization module, wherein the data verification module performs data verification on task data which is connected and transmitted to an EAM; the task scheduling module and the data synchronization module are connected with the data verification module, and when data passes through data verification, the task scheduling module is used for performing sequence management and distribution scheduling on data synchronization tasks transmitted to the EAM, and the data synchronization module is used for performing synchronization processing on actual data transmitted to the EAM.
The data checking module comprises a batch item checking module and a cache mechanism module, wherein the batch checking module performs traversal by taking each node transmitted to the EAM data as an independent unit to realize batch checking on the data; the cache mechanism module can self-define the updated data of the data cache pool, and the problems of data throughput and network delay caused by distance in the data verification process of the batch verification module are solved.
The data checking module also comprises a single field item checking module and a data dictionary loading module, the batch item checking module can call the data dictionary loading module to sequentially traverse the field items, the repeated items and the equipment codes, and once the checking fails, the batch checking problem is recorded and the single field item module is not called any more; the single field item checking module can check whether the field value in the data is in a data list or not; the single-field checking module can be connected with a database through the data dictionary loading module to inquire and pull all data dictionaries relevant to the selection items.
The task scheduling module comprises a task distribution module and a task execution strategy module, wherein the task distribution module can release synchronous tasks received by the external interface to a task center and sequence the synchronous tasks according to time so as to execute the tasks in order; the task execution strategy module can perform real-time or delayed processing on the received tasks, and ensures the execution efficiency of data synchronization.
The task scheduling module also comprises a task center monitoring module which can be distributed on each task execution unit and can monitor the task center; and after the task center is broken due to an accident, the task execution unit sends an alarm mail and monitors the process, so that the task center can be automatically restarted.
The data synchronization module comprises an automatic adaptation module, a page analysis module, an assembly synchronization module and an assembly execution storage module; the automatic adaptation module automatically matches various components in the data synchronization module according to the synchronization task data to obtain the components needing to perform data synchronization, and after the modules are adapted, specific data synchronization is performed; the page analysis module can simulate user operation by adopting page analysis; the component synchronization module can simulate a user to log in basic equipment data and fill and supplement detailed information of the equipment; the component execution storage module may store the serialization of the data synchronization task results.
A distributed data transmission control method specifically comprises the following steps:
s1, carrying out data verification on the data transmitted to the EAM;
s2, synchronizing the data transmitted to the EAM, and performing order management and distribution scheduling on the tasks;
and 3, monitoring a task queue of the task center, acquiring a task list in real time, and executing the task according to a task strategy.
The specific steps of carrying out data verification on the data transmitted to the EAM are as follows: analyzing parameters of data received by an EAM system; carrying out batch item verification on the received data, and stopping subsequent single item verification after verification fails; taking each node of the received data as an independent unit, and carrying out sequential traversal check on the field item, the repeated item and the equipment code of the received data; and after the batch item is verified, verifying the data single item.
The verifying the data single item specifically comprises: the single item check of the data comprises checking a mandatory item, a dependent item, a selected item, a confirmation item and a self-defined item of the data in sequence, wherein the selected item check is to check whether a field value of the data is in a pull-down data list or not; and connecting the primary database by adopting a data dictionary loading mode, inquiring all data related to the pull option, and analyzing and setting the updating time of the data cache pool by utilizing a human-computer interaction simulation operation interval.
The specific steps of performing synchronization tasks on the data transmitted to the EAM in S2, and performing order management and allocation scheduling on the tasks include: carrying out synchronous tasks on the data transmitted to the EAM, and carrying out ordered management and distribution scheduling on the tasks; the data synchronization task transmitted to the EAM is issued to a task center of the EAM system and is executed in order according to the task sequence; and a task center monitoring module is distributed on each task executor to monitor the task center, and the task executor sends alarm information after the task center is broken due to an accident.
The invention has the following remarkable effects: the distributed data transmission control system and the method adopt the data transmission control of a first verification and then synchronization mechanism, and a user can obtain a detailed and definite field item analysis table obtained in a simulated actual operation mode only by a data verification function, so that the user is prompted in an all-round way, and the user can conveniently obtain a processing scheme according to a verification result; after verification, the user can perform steady data synchronization work in a mode of simulating actual operation only by data synchronization, and the workload of the user can be obviously liberated.
Drawings
Fig. 1 is a schematic diagram of a distributed data transmission control system according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the embodiments.
As shown in fig. 1, a distributed data transmission control system includes a data checking module, a task scheduling module, and a data synchronization module, where the data checking module includes a batch item checking module, a data dictionary loading module, a cache mechanism module, and a single-field item checking module, and the batch checking module may traverse with each node as an independent unit to implement batch checking of data; the batch item checking module can call the data dictionary loading module to perform sequential traversal checking on the field items, the repeated items and the equipment codes, records the problem of batch checking once the checking fails, and does not call the single-field item checking module to perform single item checking; the single field item checking module can check whether the field value in the data is in a data list or not, the single field item checking module can be connected with a database query through the data dictionary loading module to pull all data dictionaries related to the selection item, and the pressure of each data query on the database is reduced; the cache mechanism module can self-define the update time of the data cache pool, and solve the problems of large data quantity and long data loading time caused by network interval distance in the loading process of the data dictionary loading module; the task scheduling module is connected with the data verification module and is used for sequencing, distributing, scheduling and allocating the importance of tasks in the received data after the data verification is finished, wherein the task scheduling module comprises a task distribution module, a task execution strategy module and a task center monitoring module, and the task issuing module can issue synchronous tasks received by an external interface to a task center and perform sequencing according to time to ensure the ordered execution of the tasks; the task execution strategy module can perform real-time or delayed processing on the tasks so as to control the execution efficiency of data synchronization; the task center monitoring module can be integrated and distributed on each task execution unit to ensure the monitoring of the task center, and after the task center is broken due to accident, the task execution units send out an alarm mail and monitor the process to ensure the automatic restart of the services such as the task center and the like; the data synchronization module comprises an automatic adaptation module, a page analysis module, a component synchronization module and a component execution storage module, wherein the automatic adaptation module automatically matches each component in the data synchronization module according to specific task data of a synchronization task to obtain a component which needs to execute data synchronization, and executes specific data synchronization after the modules are adapted; the page analysis module can simulate user operation in a page analysis mode, and comprises functions of simulating user login, user logout, user click confirmation and the like, and obtains an operation sequence value of the actual operation of the user; the component synchronization module comprises a component basic operation module and a component detail operation module, the component basic operation module simulates a user to log in and enter basic equipment data, and the component detail operation module is used for filling and supplementing detailed information of the equipment data; and the component executes the serialized storage of the data synchronization task result by the storage module.
A distributed data transmission control method specifically comprises the following steps:
step 1.1, analyzing parameters of data received by an EAM system;
step 1.2, carrying out batch item verification on the received data, and stopping subsequent single item verification after verification fails;
taking each node of the received data as an independent unit, and carrying out sequential traversal check on the field item, the repeated item and the equipment code of the received data;
step 1.3, after the batch item is verified, verifying the data single item;
the single item check of the data comprises checking a mandatory item, a dependent item, a selected item, a confirmation item and a self-defined item of the data in sequence, wherein the selected item check is to check whether a field value of the data is in a pull-down data list or not; connecting a primary database by adopting a data dictionary loading mode, inquiring all data related to the pull option, and analyzing and setting the updating time of the data cache pool by utilizing a human-computer interaction simulation operation interval;
step 2.1, the data synchronization task transmitted to the EAM is issued to a task center of the EAM system and is executed in order according to the task sequence;
step 2.2, distributing a task center monitoring module on each task executor to monitor the task center, and sending alarm information by the task executor after the task center is broken by accident;
and 3, monitoring a task queue of the task center, acquiring a task list in real time, and executing the task according to a task strategy.
Claims (10)
1. A distributed data transmission control system is characterized by comprising a data verification module, a task scheduling module and a data synchronization module, wherein the data verification module performs data verification on task data which is connected with and transmitted to an EAM; the task scheduling module and the data synchronization module are connected with the data verification module, and when data passes through data verification, the task scheduling module is used for performing sequence management and distribution scheduling on data synchronization tasks transmitted to the EAM, and the data synchronization module is used for performing synchronization processing on actual data transmitted to the EAM.
2. The distributed data transmission control system according to claim 1, wherein the data checking module includes a batch item checking module and a cache mechanism module, wherein the batch checking module performs traversal with each node transmitted to the EAM data as an independent unit to realize batch checking of the data; the cache mechanism module can self-define the updated data of the data cache pool, and the problems of data throughput and network delay caused by distance in the data verification process of the batch verification module are solved.
3. The distributed data transmission control system according to claim 1, wherein the data checking module further comprises a single field item checking module and a data dictionary loading module, the batch item checking module can call the data dictionary loading module to sequentially traverse field items, repeated items and device codes, and once checking fails, a batch checking problem is recorded and the single field item module is not called any more; the single field item checking module can check whether the field value in the data is in a data list or not; the single-field checking module can be connected with a database through the data dictionary loading module to inquire and pull all data dictionaries relevant to the selection items.
4. The distributed data transmission system according to claim 1, wherein the task scheduling module comprises a task distribution module and a task execution policy module, and the task distribution module can issue the synchronous tasks received by the external interface to a task center and sequence the tasks in time order to execute the tasks in order; the task execution strategy module can perform real-time or delayed processing on the received tasks, and ensures the execution efficiency of data synchronization.
5. The distributed data transmission system according to claim 4, wherein the task scheduling module further comprises a task center monitoring module, and the task center monitoring module can be distributed on each task execution unit and can monitor the task center; and after the task center is broken due to an accident, the task execution unit sends an alarm mail and monitors the process, so that the task center can be automatically restarted.
6. The distributed data transmission system according to claim 1, wherein said data synchronization module comprises an automatic adaptation module, a page parsing module, a component synchronization module, and a component execution storage module; the automatic adaptation module automatically matches various components in the data synchronization module according to the synchronization task data to obtain the components needing to perform data synchronization, and after the modules are adapted, specific data synchronization is performed; the page analysis module can simulate user operation by adopting page analysis; the component synchronization module can simulate a user to log in basic equipment data and fill and supplement detailed information of the equipment; the component execution storage module may store the serialization of the data synchronization task results.
7. A distributed data transmission control method is characterized by specifically comprising the following steps:
s1, carrying out data verification on the data transmitted to the EAM;
s2, synchronizing the data transmitted to the EAM, and performing order management and distribution scheduling on the tasks;
and 3, monitoring a task queue of the task center, acquiring a task list in real time, and executing the task according to a task strategy.
8. The method according to claim 7, wherein the data verification of the data transmitted to the EAM comprises: analyzing parameters of data received by an EAM system; carrying out batch item verification on the received data, and stopping subsequent single item verification after verification fails; taking each node of the received data as an independent unit, and carrying out sequential traversal check on the field item, the repeated item and the equipment code of the received data; and after the batch item is verified, verifying the data single item.
9. The distributed data transmission method according to claim 8, wherein the verifying the data entry specifically includes: the single item check of the data comprises checking a mandatory item, a dependent item, a selected item, a confirmation item and a self-defined item of the data in sequence, wherein the selected item check is to check whether a field value of the data is in a pull-down data list or not; and connecting the primary database by adopting a data dictionary loading mode, inquiring all data related to the pull option, and analyzing and setting the updating time of the data cache pool by utilizing a human-computer interaction simulation operation interval.
10. The method according to claim 7, wherein the step of S2 performing synchronization task on data transmitted to the EAM, and performing order management and distribution scheduling on the task specifically comprises: carrying out synchronous tasks on the data transmitted to the EAM, and carrying out ordered management and distribution scheduling on the tasks; the data synchronization task transmitted to the EAM is issued to a task center of the EAM system and is executed in order according to the task sequence; and a task center monitoring module is distributed on each task executor to monitor the task center, and the task executor sends alarm information after the task center is broken due to an accident.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010984787.5A CN112422616A (en) | 2020-09-18 | 2020-09-18 | Distributed data transmission control system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010984787.5A CN112422616A (en) | 2020-09-18 | 2020-09-18 | Distributed data transmission control system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112422616A true CN112422616A (en) | 2021-02-26 |
Family
ID=74854916
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010984787.5A Pending CN112422616A (en) | 2020-09-18 | 2020-09-18 | Distributed data transmission control system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112422616A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| CN106789395A (en) * | 2016-11-22 | 2017-05-31 | 盐城工学院 | A kind of Distributed PDM system monitoring data transmission method based on Web |
| WO2017089355A1 (en) * | 2015-11-24 | 2017-06-01 | T2 Data Ab | Data synchronization in a distributed data storage system |
| US20190370381A1 (en) * | 2018-05-31 | 2019-12-05 | Microsoft Technology Licensing, Llc | Distributed data synchronization in a distributed computing system |
-
2020
- 2020-09-18 CN CN202010984787.5A patent/CN112422616A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| WO2017089355A1 (en) * | 2015-11-24 | 2017-06-01 | T2 Data Ab | Data synchronization in a distributed data storage system |
| CN106789395A (en) * | 2016-11-22 | 2017-05-31 | 盐城工学院 | A kind of Distributed PDM system monitoring data transmission method based on Web |
| US20190370381A1 (en) * | 2018-05-31 | 2019-12-05 | Microsoft Technology Licensing, Llc | Distributed data synchronization in a distributed computing system |
Non-Patent Citations (1)
| Title |
|---|
| 巴军等: "基于EAM系统设备数据传输的中间件研制及应用", 《武汉大学学报(工学版)》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110069572B (en) | HIVE task scheduling method, device, equipment and storage medium based on big data platform | |
| CN101707399B (en) | Method and system for acquiring electric energy information | |
| CN107291565B (en) | Operation and maintenance visual automatic operation platform and implementation method | |
| CN102880503B (en) | Data analysis system and data analysis method | |
| CN110309231B (en) | Cross-machine-room data synchronization method and system | |
| CN109192248A (en) | Biological information analysis system, method and cloud computing platform system based on cloud platform | |
| CN105677465B (en) | The data processing method and device of batch processing are run applied to bank | |
| CN107092489B (en) | Processing method and system based on application version release | |
| CN109840144B (en) | Information service scheduling method and system for cross-mechanism batch service request | |
| CN111026602A (en) | Health inspection scheduling management method and device of cloud platform and electronic equipment | |
| CN113419920B (en) | Real-time monitoring method for joint debugging test process of simulation test system of Internet of things management platform | |
| CN111913933B (en) | Power grid historical data management method and system based on unified support platform | |
| CN115687486B (en) | Light-weight data acquisition method and device based on keyle | |
| CN113703997A (en) | Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method | |
| CN102662655B (en) | The structure system and method for version | |
| CN115146000A (en) | Database data synchronization method and device, electronic equipment and storage medium | |
| CN113658351B (en) | Method and device for producing product, electronic equipment and storage medium | |
| CN111277425B (en) | Centralized data transmission management device | |
| CN112633850A (en) | Method for realizing service flow automation by managing, calling and monitoring RPA robot | |
| CN110333909B (en) | Modeling method for automatic configuration of automatic secondary equipment model | |
| CN112422616A (en) | Distributed data transmission control system and method | |
| CN111459748A (en) | Operation and maintenance control method, system and device based on command control and storage medium | |
| ZA202306517B (en) | System and method for verifying control logic design of nuclear power plant | |
| CN115509714A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN112667393B (en) | Method and device for building distributed task computing scheduling framework and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210226 |