CN114265903A - Data collection and distribution system and working method - Google Patents
Data collection and distribution system and working method Download PDFInfo
- Publication number
- CN114265903A CN114265903A CN202111621777.6A CN202111621777A CN114265903A CN 114265903 A CN114265903 A CN 114265903A CN 202111621777 A CN202111621777 A CN 202111621777A CN 114265903 A CN114265903 A CN 114265903A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- message
- receiving
- application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013480 data collection Methods 0.000 title claims abstract description 15
- 238000009826 distribution Methods 0.000 title claims abstract description 15
- 238000000034 method Methods 0.000 title claims abstract description 10
- 238000013500 data storage Methods 0.000 claims abstract description 52
- 238000004519 manufacturing process Methods 0.000 claims abstract description 39
- 238000004891 communication Methods 0.000 claims abstract description 6
- 238000005516 engineering process Methods 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 3
- 230000002688 persistence Effects 0.000 claims description 3
- 230000002085 persistent effect Effects 0.000 claims description 3
- 238000003860 storage Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims 3
- 238000012545 processing Methods 0.000 abstract description 2
- 238000011161 development Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
Images
Abstract
The invention relates to a data collecting and distributing system and a working method, belonging to the technical field of information communication and data processing. The system comprises a data production module, a data application module, a message service module, a data storage module and a data transceiving module, wherein the data production module, the data application module and the data transceiving module are all connected with the message service module, and the data transceiving module is connected with the data storage module. The invention realizes the data collection and distribution among different information systems, does not need to expose the database connection information of each other, has no data security risk and has high system security.
Description
Technical Field
The invention relates to a data collecting and distributing system and a working method, belonging to the technical field of information communication and data processing.
Background
With the development of internet technology, more and more information systems are provided, which is convenient for people to live and liberates manpower, but because of factors such as technical differences among different information systems, data is difficult to share, and a serious information isolated island problem exists.
In order to realize data sharing, a traditional technical scheme is to independently develop a data synchronization system, and realize data synchronization and sharing between different information systems at a database level, and the way is data synchronization and sharing between two systems one by one.
When data are shared among a plurality of systems, each system can provide different data content and formats, each system needs different data formats shared by other systems, data conversion involved in the data conversion is needed, service logic becomes very complex, and a very large development workload is needed. And data synchronization and sharing among a plurality of different systems need to expose database connection information of each other, so that a very serious data security risk exists.
In terms of the synchronization mode, the mode is mostly based on a database trigger, when data in one of the service databases is changed, the system actively converts and synchronizes the data to other databases needing to be shared, operations of the plurality of shared databases may be serial or parallel, in any mode, serious performance influence is generated on the service databases, and the influence is aggravated along with the increase of the shared data nodes.
Another common way is to use ETL software to implement data transformation and sharing between different systems, which reduces the development workload of the systems, but still exposes the performance of the database connection and project service system itself.
Therefore, the invention provides a method and a system for conveniently realizing data collection and distribution among different information systems, so as to realize data synchronization and sharing among different information systems.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a data collecting and distributing system and a working method thereof, which can realize the data collection and distribution among different information systems, do not need to expose the database connection information of each other, have no data security risk and have high system security.
The technical scheme of the invention is as follows:
a data collecting and distributing system comprises a data production module, a data application module, a message service module, a data storage module and a data transceiving module, wherein,
the data production module, the data application module and the data receiving and sending module are all connected with a message service module, and the data receiving and sending module is connected with a data storage module;
a message service module: the data production module, the data application module and the data transceiver module are all registered and logged on the message service module, and the message service module issues a unique number to each logged module, wherein the number is an identifier of the logged module on the message service module;
a data production module: the system is responsible for production data and updating and maintaining data;
a data receiving and transmitting module: the data storage module is responsible for centralized management of data in the system, receives the data of the data production module, stores the data into the data storage module and also can provide the data to the data application module; in the whole system, only the data interaction exists between the data transceiver module and the data storage module, data can be written, updated and deleted to the data storage module, and data can also be read from the data storage module and provided for other data application modules.
A data storage module: the module responsible for data persistence only interacts with the data transceiver module and does not log in the message service module;
a data application module: the module using data in the system is provided with an independent data storage module, all data application modules are registered on the data transceiver module, the data types required by the data application modules are declared, and the data types are stored in a configuration library of the data transceiver module; when the data transceiver module sends data, it is convenient to determine which data application modules should send messages.
Preferably, the communication technology between the data production module, the data application module, the data transceiver module and the message service module adopts a TCP protocol or a UDP protocol, and may also be technology based on other protocols.
Preferably, the data storage module uses a relational database SQL Server, MySql or Oracle.
Preferably, the data storage module uses a non-relational database, Redis, or MongoDB.
Preferably, the data storage module uses the full text search engine elastic search.
The working method of the data collecting and distributing system comprises the following steps:
step 101: the data production module generates new data through adding, updating or deleting operations;
step 102: the data production module sends a message to the data receiving and sending module through the message service module, wherein the message comprises a data production module number, a data content type, a data operation type and a data content;
step 103: after receiving the message, the data receiving and transmitting module analyzes the message content and acquires the data content, and then stores the data into the data storage module; the data storage module comprises a database, a file, a cache and the like, and can also be stored in multiple modes at the same time.
Step 104: after the data receiving and sending module finishes data storage, the data receiving and sending module reads the data from the configuration library of the data receiving and sending module to which the data should be sent, and sends a new data message by using a broadcast message, wherein the message should contain the unique identification, the data type, the operation type and other contents of the data.
Step 105: the data application module verifies the received message according to the message content, and determines whether the data is the data required by the application module, if so, proceeds to step 106.
Step 106: the data application module sends a request message for calling data to the data transceiver module.
Step 107: the data transceiver module retrieves the specified data from the data storage module.
Step 108: the data transceiver module provides data to the data application module in a feedback request message manner, and the step may be that the data transceiver module provides complete data to the data application module in a point-to-point message manner, or provides a data interface address to the data application module in a point-to-point message manner.
Step 109: the data application module receives the data and performs subsequent operations within the module, including local persistent storage of the data.
Step 110: and the data application module informs the data receiving and transmitting module of completing data receiving in a point-to-point message mode.
Step 111: the data transceiver module updates the used condition of the data, so that the statistical work of the data storage center is facilitated.
The working principle is as follows: in the system, after the data production module generates new data, a message with the type of production data is sent to the data receiving and sending module through the message service module, and after the data receiving and sending module receives the message with the type of production data, the data storage module can store the data after analyzing the message content and acquiring the data content.
After finishing data storage and data caching, the data transceiver module reads the data from the configuration library of the data transceiver module and sends the data to which data application modules. And sending a new data message by using a broadcast message, wherein the message contains the unique identification, the data type and other contents of the data, so that the data application module judges whether the data needs to be called and used. After receiving the broadcast message of the data transceiver module, the data application module judges whether the data is the data required by the module according to the data type, if the data is the data required by the module, the data application module feeds back the data required to be called to the data transceiver center, and the data transceiver module calls the data from the data storage module and feeds back the data to the data application module.
The invention has the beneficial effects that:
1. the invention provides a data collection and distribution system and a working method, which can realize data collection and distribution among different information systems without exposing mutual database connection information, have no data security risk and have high system security.
2. The invention has simple business logic and small development workload.
Drawings
FIG. 1 is a system network topology of the present invention;
FIG. 2 is a flow chart of the use of the present invention;
Detailed Description
The present invention will be further described by way of examples, but not limited thereto, with reference to the accompanying drawings.
Example 1:
as shown in fig. 1-2, the present embodiment provides a data collection and distribution system including a data production module, a data application module, a message service module, a data storage module, and a data transceiving module, wherein,
the data production module, the data application module and the data receiving and sending module are all connected with a message service module, and the data receiving and sending module is connected with a data storage module;
a message service module: the data production module, the data application module and the data transceiver module are all registered and logged on the message service module, and the message service module issues a unique number to each logged module, wherein the number is an identifier of the logged module on the message service module;
a data production module: the system is responsible for production data and updating and maintaining data;
a data receiving and transmitting module: the data storage module is responsible for centralized management of data in the system, receives the data of the data production module, stores the data into the data storage module and also can provide the data to the data application module; in the whole system, only the data interaction exists between the data transceiver module and the data storage module, data can be written, updated and deleted to the data storage module, and data can also be read from the data storage module and provided for other data application modules.
A data storage module: the module responsible for data persistence only interacts with the data transceiver module and does not log in the message service module;
a data application module: the module using data in the system is provided with an independent data storage module, all data application modules are registered on the data transceiver module, the data types required by the data application modules are declared, and the data types are stored in a configuration library of the data transceiver module; when the data transceiver module sends data, it is convenient to determine which data application modules should send messages.
The data production module, the data application module and the communication technology between the data receiving and sending module and the message service module adopt TCP protocol.
The data storage module uses a relational database SQL Server, MySql or Oracle.
The working method of the data collecting and distributing system comprises the following steps:
step 101: the data production module generates new data through adding, updating or deleting operations;
step 102: the data production module sends a message to the data receiving and sending module through the message service module, wherein the message comprises a data production module number, a data content type, a data operation type and a data content;
step 103: after receiving the message, the data receiving and transmitting module analyzes the message content and acquires the data content, and then stores the data into the data storage module; the data storage module comprises a database, a file, a cache and the like, and can also be stored in multiple modes at the same time.
Step 104: after the data receiving and sending module finishes data storage, the data receiving and sending module reads the data from the configuration library of the data receiving and sending module to which the data should be sent, and sends a new data message by using a broadcast message, wherein the message should contain the unique identification, the data type, the operation type and other contents of the data.
Step 105: the data application module verifies the received message according to the message content, and determines whether the data is the data required by the application module, if so, proceeds to step 106.
Step 106: the data application module sends a request message for calling data to the data transceiver module.
Step 107: the data transceiver module retrieves the specified data from the data storage module.
Step 108: the data transceiver module provides data to the data application module in a feedback request message manner, and the step may be that the data transceiver module provides complete data to the data application module in a point-to-point message manner, or provides a data interface address to the data application module in a point-to-point message manner.
Step 109: the data application module receives the data and performs subsequent operations within the module, including local persistent storage of the data.
Step 110: and the data application module informs the data receiving and transmitting module of completing data receiving in a point-to-point message mode.
Step 111: the data transceiver module updates the used condition of the data, so that the statistical work of the data storage center is facilitated.
The working principle is as follows: in the system, after the data production module generates new data, a message with the type of production data is sent to the data receiving and sending module through the message service module, and after the data receiving and sending module receives the message with the type of production data, the data storage module can store the data after analyzing the message content and acquiring the data content.
After finishing data storage and data caching, the data transceiver module reads the data from the configuration library of the data transceiver module and sends the data to which data application modules. And sending a new data message by using a broadcast message, wherein the message contains the unique identification, the data type and other contents of the data, so that the data application module judges whether the data needs to be called and used. After receiving the broadcast message of the data transceiver module, the data application module judges whether the data is the data required by the module according to the data type, if the data is the data required by the module, the data application module feeds back the data required to be called to the data transceiver center, and the data transceiver module calls the data from the data storage module and feeds back the data to the data application module.
Example 2:
a data collection and distribution system, constructed as described in embodiment 1, except that the communication technology between the data production module, the data application module, and the data transceiving module and the message service module employs UDP protocol, and the data storage module uses a non-relational database Redis or MongoDB.
Example 3:
a data collection and distribution system constructed as described in embodiment 1 except that the data storage module uses a full text search engine elastic search.
Claims (6)
1. A data collecting and distributing system is characterized by comprising a data production module, a data application module, a message service module, a data storage module and a data transceiving module, wherein,
the data production module, the data application module and the data receiving and sending module are all connected with a message service module, and the data receiving and sending module is connected with a data storage module;
a message service module: the data production module, the data application module and the data receiving and sending module are all registered and logged on the message service module, the message service module sends a unique number to each logged module, and the number is an identifier of the logged module on the message service module;
a data production module: the system is responsible for production data and updating and maintaining data;
a data receiving and transmitting module: the data storage module is responsible for centralized management of data in the system, receives the data of the data production module, stores the data into the data storage module and also can provide the data to the data application module;
a data storage module: the module responsible for data persistence only interacts with the data transceiver module and does not log in the message service module;
a data application module: the data transmission and receiving module is provided with an independent data storage module, all data application modules are registered on the data transmission and receiving module, data types required by the data application modules are declared, and the data types are stored in a configuration library of the data transmission and receiving module.
2. The data collection and distribution system of claim 1, wherein the communication technology between the data production module, the data application module, and the data transceiver module and the message service module employs a TCP protocol or a UDP protocol.
3. The data collection and distribution system of claim 1, wherein the data storage module uses a relational database SQL Server, MySql, or Oracle.
4. The data collection and distribution system of claim 1, wherein the data storage module uses a non-relational database, Redis, or MongoDB.
5. The data collection and distribution system of claim 1, wherein the data storage module uses a full text search engine elastic search.
6. A method of operating a data collection and distribution system according to claim 1, characterized by the steps of:
step 101: the data production module generates new data through adding, updating or deleting operations;
step 102: the data production module sends a message to the data receiving and sending module through the message service module, wherein the message comprises a data production module number, a data content type, a data operation type and a data content;
step 103: after receiving the message, the data receiving and transmitting module analyzes the message content and acquires the data content, and then stores the data into the data storage module;
step 104: after the data receiving and sending module finishes data storage, reading data from a configuration library of the data receiving and sending module to which data application modules the data should be sent, and sending a new data message by using a broadcast message;
step 105: the data application module verifies the received message according to the message content, and judges whether the data is the data required by the application module, if so, the step 106 is performed;
step 106: the data application module sends a request message for calling data to the data transceiver module;
step 107: the data receiving and transmitting module calls the appointed data from the data storage module;
step 108: the data receiving and sending module provides data to the data application module in a mode of feeding back the request message, and the step can be that the data receiving and sending module provides complete data to the data application module in a point-to-point message mode, or provides a data interface address to the data application module in a point-to-point message mode;
step 109: the data application module receives the data and carries out subsequent operation in the module, wherein the subsequent operation comprises local persistent storage of the data;
step 110: the data application module informs the data receiving and transmitting module of completing data receiving in a point-to-point message mode;
step 111: the data transceiver module updates the used condition of the data, so that the statistical work of the data storage center is facilitated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111621777.6A CN114265903B (en) | 2021-12-28 | 2021-12-28 | Data collection and distribution system and working method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111621777.6A CN114265903B (en) | 2021-12-28 | 2021-12-28 | Data collection and distribution system and working method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114265903A true CN114265903A (en) | 2022-04-01 |
| CN114265903B CN114265903B (en) | 2024-04-30 |
Family
ID=80830700
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111621777.6A Active CN114265903B (en) | 2021-12-28 | 2021-12-28 | Data collection and distribution system and working method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114265903B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013151529A2 (en) * | 2012-04-02 | 2013-10-10 | Basil Nasri Tawfig Moshtaha | Method and system for transmitting and/or receiving advertisement and data contents and providing multiple value added services on electronic communication device with a display mechanism |
| CN110413677A (en) * | 2019-07-30 | 2019-11-05 | 无锡柠檬科技服务有限公司 | A kind of Distributed Data Synchronization method and system for supporting concurrently to apply |
| CN110704192A (en) * | 2019-09-30 | 2020-01-17 | 的卢技术有限公司 | Diversified data cloud storage method and system |
| CN111177246A (en) * | 2019-12-27 | 2020-05-19 | 腾讯云计算（北京）有限责任公司 | Service data processing method and device |
| CN113742400A (en) * | 2021-09-10 | 2021-12-03 | 哈尔滨工业大学（威海） | Network data acquisition system and method based on self-adaptive constraint conditions |
-
2021
- 2021-12-28 CN CN202111621777.6A patent/CN114265903B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013151529A2 (en) * | 2012-04-02 | 2013-10-10 | Basil Nasri Tawfig Moshtaha | Method and system for transmitting and/or receiving advertisement and data contents and providing multiple value added services on electronic communication device with a display mechanism |
| CN110413677A (en) * | 2019-07-30 | 2019-11-05 | 无锡柠檬科技服务有限公司 | A kind of Distributed Data Synchronization method and system for supporting concurrently to apply |
| CN110704192A (en) * | 2019-09-30 | 2020-01-17 | 的卢技术有限公司 | Diversified data cloud storage method and system |
| CN111177246A (en) * | 2019-12-27 | 2020-05-19 | 腾讯云计算（北京）有限责任公司 | Service data processing method and device |
| CN113742400A (en) * | 2021-09-10 | 2021-12-03 | 哈尔滨工业大学（威海） | Network data acquisition system and method based on self-adaptive constraint conditions |
Non-Patent Citations (1)
| Title |
|---|
| 陈宇收;: "基于Mycat的分布式数据存储研究", 中国新通信, no. 22, 20 November 2018 (2018-11-20) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114265903B (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104333512B (en) | A kind of distributed memory database accesses system and method | |
| CN105404701A (en) | Peer-to-peer network-based heterogeneous database synchronization method | |
| CN110912980B (en) | Order state synchronization method, system and storage medium | |
| US20030018701A1 (en) | Peer to peer collaboration for supply chain execution and management | |
| CN106294741B (en) | A kind of automated data inquiry synchronization storage system | |
| CN109923847A (en) | Call discovery method, apparatus, equipment and the storage medium of link | |
| CN112445863A (en) | Real-time data synchronization method and system | |
| CN109815294A (en) | A kind of dereliction Node distribution parallel data storage method and system | |
| CN101681461B (en) | The equipment of managing electronic communities of users and method | |
| CN104536965A (en) | System and method for data query and presentation under big data condition | |
| CN104468299A (en) | Enterprise service bus system based on user rule | |
| US8054764B2 (en) | Transmitting critical table information in databases | |
| CN111858722A (en) | Big data application system and method based on Internet of things | |
| CN102262561B (en) | The dispatching method that massive tasks of databases processes | |
| CN114265903A (en) | Data collection and distribution system and working method | |
| CN109634757B (en) | Data acquisition method for seismic industry big data processing | |
| US20210152651A1 (en) | Method for the event-controlled retrieval of process data | |
| CN114531479B (en) | General data acquisition system based on MQTT | |
| CN115344633A (en) | Data processing method, device, equipment and storage medium | |
| CN109039427A (en) | A kind of ship monitor managing device | |
| KR20080029362A (en) | Mediation apparatus processing interface between client and server and interface method thereof | |
| CN112104888B (en) | Method and system for grouping live broadcast users | |
| CN108347373B (en) | Method and system for realizing instant messaging of enterprise APP | |
| CN114063931A (en) | Data storage method based on big data | |
| CN113220732A (en) | Data matching method and system for communication sharing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |