CN116260893B - Message subscription and publishing device of data processing system - Google Patents
Message subscription and publishing device of data processing system Download PDFInfo
- Publication number
- CN116260893B CN116260893B CN202310068318.2A CN202310068318A CN116260893B CN 116260893 B CN116260893 B CN 116260893B CN 202310068318 A CN202310068318 A CN 202310068318A CN 116260893 B CN116260893 B CN 116260893B
- Authority
- CN
- China
- Prior art keywords
- message
- module
- mode
- data processing
- ipc
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/26—Special purpose or proprietary protocols or architectures
Abstract
The application discloses a data processing system message subscription and release device, wherein the data processing system comprises a plurality of nodes, at least one data processing service is operated on each node, the data processing system comprises at least one node provided with a message proxy module, each data processing service is provided with a message subscription module and a message release module, and the message subscription module subscribes to messages in an IPC mode or a TCP mode; the message publishing module publishes the message in an IPC or TCP mode; the message proxy module subscribes to messages published by other nodes in a TCP mode and publishes the messages in an IPC mode; subscribing the information published by the node in an IPC mode and publishing the information in TCP and IPC modes. The application has reasonable design, adopts TCP and IPC modes to subscribe and publish, reduces the data volume transmitted by the TCP network in the system and improves the communication efficiency.
Description
Technical Field
The application belongs to the technical field of subscription and publication of data processing systems, and particularly relates to a message subscription and publication device of a data processing system.
Background
To ensure reliability of data processing, conventional data processing systems often employ a node-centric active-standby mode. Along with the rapid rise of the scale and complexity of data processing, the defects of the traditional centralized data processing system are developed gradually, the function of a standby machine is not fully exerted, the system is involuntarily controllable, the expansibility is not high, the deployment is not flexible enough, and the development requirements of more and more complex and diversified data processing cannot be met.
In order to overcome the shortcomings of conventional centralized data processing system architectures, the data processing system may be designed as a distributed architecture, and the system may be functionally split into multiple data processing services that operate in a clustered environment. These data processing services operating in a clustered environment require a large amount of real-time data interaction within the cluster. There are many ways of data interaction within the cluster, such as the traditional TCP way, the message middleware way, etc. If there are a large number of data processing services included in a system, all data processing services need to be connected to each other in pairs to form a complex mesh network for such a system. For complex mesh networks, the data is subscribed and published in a TCP mode, and the efficiency is low.
Disclosure of Invention
Aiming at the defects in the prior art, the application provides a message subscribing and publishing device of a data processing system, which has reasonable design, converts a complex mesh network for subscribing and publishing data into an internal and external structure, adopts TCP and IPC modes for subscribing and publishing, reduces the data volume transmitted by the TCP network in the system and improves the communication efficiency.
In order to solve the technical problems, the application adopts the following technical scheme: a data processing system message subscription publishing device, the data processing system comprising a plurality of nodes, each node running at least one data processing service thereon, characterized by: at least one node is provided with a message proxy module, and each data processing service is provided with a message subscription module and a message publishing module;
the message subscription module subscribes messages in an IPC mode or a TCP mode, deletes the theme mark of the received messages to obtain PDU, and then decodes the PDU to generate data to be provided for the data processing service;
the message issuing module encodes the data operated by each data processing service on each node into PDU, processes the theme mark, combines the theme mark and the PDU into a message and issues the message in an IPC or TCP mode;
the message proxy module subscribes to the messages published by the message proxy modules or the message publishing modules of other nodes in a TCP mode and publishes the messages in an IPC mode; subscribing the information released by the information release module in the node in an IPC mode and releasing the information in a TCP and IPC mode;
the IPC method is ZMQ IPC method, and the TCP method is ZMQ TCP method.
The above-mentioned data processing system message subscription and release device is characterized in that: the message subscription module comprises:
the first external interface module provides a subscription, logout, start and stop interface;
the first theme processing module generates a theme name according to subscription information corresponding to the data required by subscription;
the topic management module is used for managing subscribed topic identifiers, and if the subscribed topic list does not have the topic identifier, the topic management module adds the topic identifier into the subscribed topic list and subscribes to the topic identifier;
the data receiving module subscribes to receive the message in a TCP mode or an IPC mode, and deletes the theme mark of the received message to obtain PDU;
a decoding module for decoding the PDU into data and providing the data to a data processing service for processing; the decoding method adopted by the decoding module is a basic coding rule ASN.1-BER.
The above-mentioned data processing system message subscription and release device is characterized in that: in the first step, the message is subscribed by an IPC mode or a TCP mode, specifically:
if the message subscription module is located on the node where the message agent module exists, the message subscription module subscribes to the message in an IPC mode; otherwise, the message is subscribed by using a TCP mode.
The above-mentioned data processing system message subscription and release device is characterized in that: the message issuing module comprises:
a second external interface module providing a send, start, stop interface;
the coding module codes the data operated by the data processing service into PDU; wherein, the coding module adopts a distinguishable coding rule ASN.1-DER coding;
the second theme processing module generates a theme name according to the data operated by the data processing service;
and the data sending module combines the PDU generated by the encoding module and the theme name generated by the second theme processing module into a message and issues the message in an IPC mode or a TCP mode.
The above-mentioned data processing system message subscription and release device is characterized in that: step one, a message is issued through an IPC mode or a TCP mode, which specifically comprises the following steps:
if the message publishing module is located on the node where the message proxy module is located, the message publishing module publishes the message in an IPC mode; otherwise, the message is issued by using a TCP mode.
The above-mentioned data processing system message subscription and release device is characterized in that: the nodes are x 86-64, loongson MIPS64, and arm system PC, industrial computer, server, notebook computer or singlechip.
The above-mentioned data processing system message subscription and release device is characterized in that: the message broker module comprises:
the internal message subscription module is used for subscribing and receiving the messages published by the message publishing module in the node in an IPC mode and sending the messages to the message forwarding module through an internal message inproc channel;
the external message subscription module is used for subscribing the messages issued by other configured nodes through the TCP in a TCP mode and sending the messages to the message forwarding module through an external message inproc channel;
the message forwarding module forwards the message subscribed by the internal message inproc channel to the IPC channel in an IPC mode and forwards the message to the addresses and ports of all other configured nodes in a TCP mode; the information subscribed by the external information inproc channel is forwarded to the IPC channel in an IPC mode;
and the monitoring module is used for monitoring the state of the message proxy module.
The above-mentioned data processing system message subscription and release device is characterized in that: the monitoring module comprises:
the heartbeat message sending module is used for sending heartbeat messages at fixed time through the IPC channel;
the heartbeat message receiving module subscribes to the heartbeat message through the IPC channel;
the state release module is used for writing the state information of the message proxy module into the distributed cache service at regular time;
and the state maintenance module is used for reading the state information of all configured message proxy modules from the distributed cache service, acquiring heartbeat messages from the heartbeat message receiving module, and restarting the message proxy module if the state information of one message proxy module is normal but the heartbeat information cannot be continuously received.
Compared with the prior art, the application has the following advantages:
1. the application sets a message proxy module, subscribes to the messages published by other node message proxy modules or message publishing modules in a TCP mode and publishes the messages in an IPC mode; the information released by the information release module in the node is subscribed by the IPC mode and released by the TCP and IPC modes, so that the complex mesh network is converted into the internal and external structures of the node and other nodes.
2. According to the application, the message subscription module subscribes in an IPC mode or a TCP mode, and the message publishing module publishes in the IPC mode or the TCP mode, so that subscription and publishing in the TCP and IPC modes are realized, the data volume transmitted in the system through a TCP network is reduced, and the communication efficiency is improved.
In summary, the application has reasonable design, converts the complex mesh network of data subscription and release into an internal and external structure, adopts TCP and IPC modes to subscribe and release, reduces the data volume transmitted by the TCP network in the system, and improves the communication efficiency.
The technical scheme of the application is further described in detail through the drawings and the embodiments.
Drawings
Fig. 1 is a block diagram of an apparatus according to embodiment 1 of the present application.
Fig. 2 is a block diagram of the apparatus of embodiment 2 of the present application.
Fig. 3 is a block diagram of a message subscription module of the present application.
Fig. 4 is a block diagram of a message publishing module of the present application.
Fig. 5 is a block diagram of a message broker module of the present application.
Detailed Description
Example 1
A data processing system message subscription publishing device as shown in figure 1, the device comprising:
a network switch, and nodes 1, 2, 3. Respectively operating a message proxy module-1, a message proxy module-2 and a message proxy module-3 in the node 1, the node 2 and the node 3; the data processing service 1-1 and the data processing service 2-1 are operated in the node 1, the data processing service 1-2 and the data processing service 2-2 are operated in the node 2, and the data processing service 1-3 and the data processing service 2-3 are operated in the node 3. The data processing service 1-1, the data processing service 1-2 and the data processing service 1-3 form a service cluster, the data processing service 2-1, the data processing service 2-2 and the data processing service 2-3 form another service cluster, and both service clusters are part of the data processing system.
The data processing service 1-1 comprises a message subscription module 1-1 and a message publishing module 1-1, and the data processing service 2-1 comprises a message subscription module 2-1 and a message publishing module 2-1;
the data processing service 1-2 comprises a message subscription module 1-2 and a message publishing module 1-2, and the data processing service 2-2 comprises a message subscription module 2-2 and a message publishing module 2-2;
the data processing service 1-3 comprises a message subscription module 1-3 and a message publishing module 1-3, and the data processing service 2-3 comprises a message subscription module 2-3 and a message publishing module 2-3.
In this embodiment, the message proxy module-1 subscribes to all the messages published by the message proxy module-2 and the message proxy module-3 in a TCP manner, and publishes the messages published by the message proxy module-2 and the message proxy module-3 in an IPC manner; the message agent module-1 subscribes to all messages published by the message publishing module 1-1 and the message publishing module 2-1 in an IPC mode, the message agent module-1 publishes the messages received from the message publishing module 1-1 and the message publishing module 2-1 in a TCP mode and an IPC mode respectively, and the message subscribing module 1-1 and the message subscribing module 2-1 subscribe to the data published by the message agent module-1 in an IPC mode;
the message proxy module-2 subscribes to all messages published by the message proxy module-1 and the message proxy module-3 in a TCP mode, and publishes the messages published by the message proxy module-1 and the message proxy module-3 in an IPC mode, the message proxy module-2 subscribes to all messages published by the message publishing module 1-2 and the message publishing module 2-2 in an IPC mode, and the message proxy module-2 publishes the messages received from the message publishing module 1-2 and the message publishing module 2-2 in a TCP mode and an IPC mode respectively, and the message subscription module 1-2 and the message subscription module 2-2 subscribe to data published by the message proxy module-2 in an IPC mode;
the message proxy module-3 subscribes to all messages published by the message proxy module-1 and the message proxy module-2 in a TCP mode, and publishes the messages published by the message proxy module-1 and the message proxy module-2 in an IPC mode, the message proxy module-3 subscribes to all messages published by the message publishing module 1-3 and the message publishing module 2-3 in an IPC mode, and the message proxy module-3 publishes the messages received from the message publishing module 1-3 and the message publishing module 2-3 in a TCP and an IPC mode respectively, and the message subscription module 1-3 and the message subscription module 2-3 subscribe to data published by the message proxy module-3 in an IPC mode.
In this embodiment, the message publishing module 1-1 publishes a message corresponding to data running inside the data processing service 1-1 in an IPC manner; the message publishing module 2-1 publishes messages corresponding to the data running in the data processing service 2-1 in an IPC mode;
the message publishing module 1-2 publishes the message corresponding to the internal operation data of the data processing service 1-2 in an IPC mode; the message publishing module 2-2 publishes the message corresponding to the internal operation data of the data processing service 2-2 in an IPC mode;
the message publishing module 1-3 publishes the message corresponding to the internal operation data of the data processing service 1-3 in an IPC mode; the message publishing module 2-3 publishes the message corresponding to the internal operation data of the data processing service 2-3 in an IPC mode.
In this embodiment, the node 1, the node 2, and the node 3 are devices such as x86_64, MIPS64, and ARM which are servers of architecture, industrial personal computers, PCs, notebook computers, and singlechips, and the operating system running on the devices is a Linux operating system capable of supporting the architecture.
In this embodiment, it should be noted that the number of nodes in the data processing system in fig. 1, and the number of data processing services running on each node are schematic, and the data processing services running on each node need not be identical. The message publishing module and the message subscribing module in the data processing service instance are not always necessary, if the service instance only needs to send data and not receive data, the message subscribing module is not needed, and if the service instance only receives data and not send data, the message publishing module is not needed. According to actual needs, there may be any number of nodes, and the data processing service running on each node may be flexible.
In this embodiment, the messages include a data content type message and a non-data content type message, where the non-data content type message includes log information, a communication state, a task state, a control command, a service state, and a system running state.
Example 2
As shown in fig. 2, the data processing system the device comprises a network switch, and nodes 1, 2, 3. The node 1 and the node 2 respectively run a message proxy module 1 and a message proxy module 2. The data processing service 1-1 and the data processing service 2-1 are operated in the node 1, the data processing service 1-2 and the data processing service 2-2 are operated in the node 2, and the data processing service 3 is operated in the node 3. The data processing services 1-1, 1-2 form a service cluster, the data processing services 2-1, 2-2 form another service cluster, both service clusters and the data processing service 3 are part of a data processing system. The data processing service 1-1 comprises a message subscription module 1-1 and a message publishing module 1-1, and the data processing service 2-1 comprises a message subscription module 2-1 and a message publishing module 2-1;
the data processing service 1-2 comprises a message subscription module 1-2 and a message publishing module 1-2; the data processing service 2-2 includes a message subscription module 2-2 and a message publishing module 2-2. The data processing service 3 includes a message subscription module 3 and a message publishing module 3.
In this embodiment, unlike the embodiment, there is no message broker module in the node 3.
In this embodiment, the message proxy module 1 subscribes to all the messages published by the message proxy module 2 and the message publishing module 3 in a TCP manner, and publishes the messages published by the message proxy module 2 and the message publishing module 3 in an IPC manner;
the message proxy module 1 subscribes to all messages published by the message publishing module 1-1 and the message publishing module 2-1 in an IPC mode, the message proxy module 1 publishes the messages received from the message publishing module 1-1 and the message publishing module 2-1 respectively through TCP and IPC, and the message subscribing module 1-1 and the message subscribing module 2-1 subscribe to the data published by the message proxy module 1 in an IPC mode;
the message proxy module 2 subscribes to all messages published by the message proxy module 1 and the message publishing module 3 in a TCP mode, and publishes the messages published by the message proxy module 1 and the message publishing module 3 in an IPC mode, the message proxy module 2 subscribes to all the messages published by the message publishing module 1-2 and the message publishing module 2-2 in an IPC mode, and the message proxy module 2 subscribes to the data published by the message proxy module 2 in a TCP and IPC mode respectively from the messages received by the message publishing module 1-2 and the message publishing module 2-2;
the message subscription module 3 subscribes all messages published by the message proxy module 1 and the message proxy module 2 in a TCP mode, and the message publishing module 3 publishes messages corresponding to the internal operation data of the data processing service 3 in the TCP mode.
Example 3
As shown in fig. 3, in this embodiment, the message subscription module includes a first external interface module (303), a first theme processing module (304), a theme management module (305), a data receiving module (306), and a decoding module (307), the message subscription module (301) may be implemented as a dynamic library, the data processing service calls the dynamic library, and the data processing application program (302) is a component of an application message in the data processing service.
In this embodiment, the first external interface module (303) provides a subscription, logoff, start, stop interface.
In this embodiment, when the message subscription module (303) subscribes to the required data, the first external interface module (303) acquires subscription information corresponding to the subscribed data through the subscription interface, and sends the subscription information to the first theme processing module (304); wherein the subscription information is one or more integers; when the data required by subscription is log information, communication state, task state, control command, service state and system running state, the subscription information is a 4-byte integer; when the data needed by subscription is data content, the subscription information is a plurality of 4-byte integers, the plurality of 4-byte integers are respectively 4-byte integers corresponding to the data type, the data receiving and transmitting attribute, the task identifier, the equipment identifier, the data destination identifier and the data processing identifier, and if the subscription interface does not provide subscription information corresponding to the data type, the data receiving and transmitting attribute, the task identifier, the equipment identifier, the data source identifier, the data destination identifier and the data processing identifier, the subscription information parameter defaults to-1.
In the embodiment, a first topic processing module (304) generates a topic name according to subscription information; the topic name is a character string, and for log information, communication state, task state, control command, service state and system running state, the generated character string is an 8-byte character string with 16-system integers as contents; for data content, the generated character string consists of data type, data receiving and transmitting attribute, task identification, equipment identification, data source identification, data destination identification and data processing identification information, wherein each information is an 8-byte character string with 16-system integers as the content, and the information is separated by's;
the first topic processing module (304) transmits the processed topic name to the topic management module (305), the topic management module (305) judges that if the topic name is subscribed, the topic name is not processed, if the topic name is not subscribed, the topic name is subscribed, and the topic name is sent to the data receiving module (306). Only the subscribed topic names are received by the data receiving module (306).
In this embodiment, the data receiving module (306) receives the message in a TCP or IPC manner, deletes the theme mark of the received message, and then obtains a PDU, and sends the PDU to the decoding module (307), and the decoding module (307) generates data after decoding by the basic coding rule asn.1-BER, and then sends the data to the data processing application (302) for data processing.
In this embodiment, the logout interface is used to logout unwanted topic names that have been subscribed to.
In this embodiment, the first external interface module (303) sends the logout information to the first theme processing module (304) through the logout interface, the first theme processing module (304) processes the theme name according to the logout information, the processing method is the same as that described above, the first theme processing module (304) transmits the processed theme name to the theme management module (305), the theme management module (305) determines that if the theme name is an unsubscribed theme name, the processing is not performed, and if the theme name is a subscribed theme name, the theme name is logged out. If the data can be received originally, the data can not be received after logging off.
In this embodiment, the first external interface module (303) is used to control the data receiving module (306) to connect to either the ipc:///tmp/mqproxy.ipc or the already configured TCP address and port through the start-up interface. If a message proxy service module exists on the node where the data processing application (302) operates, the data receiving module (306) subscribes in an IPC mode; otherwise, subscribing in a TCP mode.
In this embodiment, the first external interface module (303) is configured to control the data receiving module (306) to close the connection to ipc:///tmp/mqproxy.
Example 4
As shown in fig. 4, in this embodiment, the message publishing module includes a second external interface module (403), an encoding module (404), a second theme processing module (405), and a data sending module (406), where the message publishing module may be implemented as a dynamic library, the data processing service invokes the dynamic library, and the data processing application (402) is a component of an application message in the data processing service.
In this embodiment, the second external interface module (403) provides a send, start, stop interface.
In this embodiment, the second external interface module (403) is used to control the data sending module (406) to connect to either the ipc:///tmp/mqproxy.ipc or the already configured TCP address and port through the start-up interface. If a message proxy service module exists on the node where the data processing application program (402) runs, the message publishing module publishes the message in an IPC mode; otherwise, the data is issued in a TCP mode.
In this embodiment, the second external interface module (403) is configured to control the data sending module (406) to close the connection to ipc:///tmp/mqproxy.
In this embodiment, the second external interface module (403) transmits the data operated by the data processing application program (402) to the encoding module (404) and the second theme processing module (405) through the transmission interface, and the encoding module (404) generates a protocol data unit PDU by encoding with a distinguishable encoding rule (asn.1-DER) and transmits the protocol data unit PDU to the second theme processing module (405); the second theme processing module (405) generates a theme name according to the data operated by the data processing application program (402), the method generates the theme name according to the subscription information, the second theme processing module (405) transmits the theme name and the PDU to the data sending module (406) at the same time, and the data sending module (406) combines the theme name and the PDU into a message and then issues the message in an IPC mode or a TCP mode.
Example 5
As shown in fig. 5, in this embodiment, the message broker module includes an internal message subscription module (501), an external message subscription module (502), a message forwarding module (503), and a monitoring module (504), where the monitoring module (504) includes a heartbeat message sending module (505), a status publishing module (506), a heartbeat message receiving module (507), and a status maintenance module (508). The message broker module may be implemented as a stand-alone process that operates as a system service, which may be referred to as a message broker service. The heartbeat message sending module (505) can be independently implemented, a message publishing module can be used, and the heartbeat message receiving module (507) can be independently implemented, and a message subscribing module can be used. The distributed caching service (509) is an external service that the message broker service needs to publish its own state and read its own state.
In this embodiment, the internal message subscription module: the message forwarding module is used for subscribing and receiving messages which are issued by all message issuing modules in the node in an IPC mode and sending the messages to the message forwarding module through an internal message inproc channel;
an external message subscription module: the method is used for subscribing the information issued by other configured nodes through TCP in a TCP mode and sending the information to the information forwarding module through an external information inproc channel;
message forwarding module: the message subscribed by the internal message inproc channel is forwarded to the IPC channel in an IPC mode and to the addresses and ports of all other configured nodes in a TCP mode; the information subscribed by the external information inproc channel is forwarded to the IPC channel in an IPC mode;
and a monitoring module: for monitoring the status of the message broker module.
In this embodiment, the heartbeat message module (505) in the monitoring module (504) sends heartbeat messages periodically through the IPC channel, the node internal message subscription module (501) subscribes to receive the heartbeat messages sent by the node heartbeat message module (505), and the external message subscription module (502) subscribes to receive the heartbeat messages sent by the configured heartbeat message modules (505) of other nodes;
the heartbeat messages received by the internal message subscription module (501) and the external message subscription module (502) enable the message forwarding module (503) to forward through the ipc:///tmp/Mqproxy.ipc channel, and the heartbeat message receiving module (507) receives the heartbeat messages so as to ensure that the message subscription and publishing device of the data processing system is in a normal state.
In this embodiment, the state publishing module (506) publishes the state information of the message broker module through the distributed cache service (509), where the state information includes time, an internal information subscription module state, an external information subscription module state, a message forwarding module state, and a universal information identifier UUID.
In this embodiment, the state maintenance module (508) reads the state information of all the message broker modules from the distributed cache service (509) and acquires heartbeat information from the heartbeat receiving module (507). If all the state information of one message proxy module is normal, but heartbeat information cannot be continuously received, restarting the message proxy module.
In this embodiment, with the current manner of subscribing with TCP, if the subscriber is already connected to the publisher, in this case, the publisher is restarted for some reason, the subscriber may not receive the data, which may affect the reliability of the data processing system. Therefore, the message proxy module is provided with the monitoring module, and the abnormal automatic recovery is carried out by monitoring the heartbeat message and the state information simultaneously, so that the automatic recovery is noninductive to the service of the data processing system, and the reliability of the system is improved.
In this embodiment, the distributed cache service (509) adopts a Redis distributed memory database, belongs to a part of a message subscription and release device of a data processing system, and can operate on the node 1, the node 2 and the node 3, or independently set a node to operate the distributed cache service (509).
The above-referenced embodiments may be implemented as computer software components or applications according to embodiments of the present application. All or part of the present application may be implemented as part of a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods or solutions in accordance with the present application by way of operation of the computer.
It will be evident to those skilled in the art that the application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The embodiments are therefore to be considered in all respects as illustrative and not restrictive. The scope of the application is indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Claims (6)
1. A data processing system message subscription publishing device, the data processing system comprising a plurality of nodes, each node running at least one data processing service thereon, characterized by: at least one node is provided with a message proxy module, and each data processing service is provided with a message subscription module and a message publishing module;
the message subscription module subscribes messages in an IPC mode or a TCP mode, deletes the topic identification of the received messages to obtain PDU, and then decodes the PDU to generate data to be provided for the data processing service;
the message issuing module encodes the data operated by each data processing service on each node into PDU, processes the theme mark, combines the theme mark and the PDU into a message and issues the message in an IPC or TCP mode;
the message proxy module subscribes to the messages published by the message proxy modules or the message publishing modules of other nodes in a TCP mode and publishes the messages in an IPC mode; subscribing the information released by the information release module in the node in an IPC mode and releasing the information in a TCP and IPC mode;
the IPC mode is ZMQ, and the TCP mode is ZMQ;
the message subscription module comprises:
the first external interface module provides a subscription, logout, start and stop interface;
the first theme processing module generates a theme name according to subscription information corresponding to the data required by subscription;
the topic management module is used for managing subscribed topic identifiers, and if the topic identifiers are not in the subscribed topic list, the topic management module adds the topic identifiers into the subscribed topic list and subscribes the topic identifiers;
the data receiving module subscribes to receive the message in a TCP mode or an IPC mode, and deletes the topic identification of the received message to obtain PDU;
a decoding module for decoding the PDU into data and providing the data to a data processing service for processing; the decoding method adopted by the decoding module is a basic coding rule ASN.1-BER;
the message issuing module comprises:
a second external interface module providing a send, start, stop interface;
the coding module codes the data operated by the data processing service into PDU; wherein, the coding module adopts a distinguishable coding rule ASN.1-DER coding;
the second theme processing module generates a theme name according to the data operated by the data processing service;
and the data sending module combines the PDU generated by the encoding module and the theme name generated by the second theme processing module into a message and issues the message in an IPC mode or a TCP mode.
2. A data processing system message subscription issuing apparatus according to claim 1, wherein: subscribing to the message by IPC mode or TCP mode, specifically:
if the message subscription module is located on the node where the message agent module exists, the message subscription module subscribes to the message in an IPC mode; otherwise, the message is subscribed by using a TCP mode.
3. A data processing system message subscription issuing apparatus according to claim 1, wherein: the message is issued by an IPC mode or a TCP mode, specifically:
if the message publishing module is located on the node where the message proxy module is located, the message publishing module publishes the message in an IPC mode; otherwise, the message is issued by using a TCP mode.
4. A data processing system message subscription issuing apparatus according to claim 1, wherein: the nodes are x 86-64, loongson MIPS64, and arm system PC, industrial computer, server, notebook computer or singlechip.
5. A data processing system message subscription issuing apparatus according to claim 1, wherein: the message broker module comprises:
the internal message subscription module is used for subscribing and receiving the messages published by the message publishing module in the node in an IPC mode and sending the messages to the message forwarding module through an internal message inproc channel;
the external message subscription module is used for subscribing the messages issued by other configured nodes through the TCP in a TCP mode and sending the messages to the message forwarding module through an external message inproc channel;
the message forwarding module forwards the message subscribed by the internal message inproc channel to the IPC channel in an IPC mode and forwards the message to the addresses and ports of all other configured nodes in a TCP mode; the information subscribed by the external information inproc channel is forwarded to the IPC channel in an IPC mode;
and the monitoring module is used for monitoring the state of the message proxy module.
6. A data processing system message subscription issuing device according to claim 5, characterized in that: the monitoring module comprises:
the heartbeat message sending module is used for sending heartbeat messages at fixed time through the IPC channel;
the heartbeat message receiving module subscribes to the heartbeat message through the IPC channel;
the state release module is used for writing the state information of the message proxy module into the distributed cache service at regular time;
and the state maintenance module is used for reading the state information of all configured message proxy modules from the distributed cache service, acquiring heartbeat messages from the heartbeat message receiving module, and restarting the message proxy module if the state information of one message proxy module is normal but the heartbeat information cannot be continuously received.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310068318.2A CN116260893B (en) | 2023-02-06 | 2023-02-06 | Message subscription and publishing device of data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310068318.2A CN116260893B (en) | 2023-02-06 | 2023-02-06 | Message subscription and publishing device of data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116260893A CN116260893A (en) | 2023-06-13 |
| CN116260893B true CN116260893B (en) | 2023-09-12 |
Family
ID=86682009
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310068318.2A Active CN116260893B (en) | 2023-02-06 | 2023-02-06 | Message subscription and publishing device of data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116260893B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104639597A (en) * | 2013-11-08 | 2015-05-20 | 塔塔咨询服务有限公司 | System(s) and method(s) for multiple sender support in low latency fifo messaging using tcp/ip protocol |
| US10785296B1 (en) * | 2017-03-09 | 2020-09-22 | X Development Llc | Dynamic exchange of data between processing units of a system |
| CN112738190A (en) * | 2020-12-24 | 2021-04-30 | 湖南博匠信息科技有限公司 | RapidIO communication dynamic management method and system |
| CN113377091A (en) * | 2021-08-12 | 2021-09-10 | 星河动力(北京)空间科技有限公司 | Carrier rocket test launch control system |
| CN114003413A (en) * | 2021-12-30 | 2022-02-01 | 深圳佑驾创新科技有限公司 | IPC data communication method, device, equipment and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8363572B2 (en) * | 2010-04-19 | 2013-01-29 | Ipc Systems, Inc. | Session initiation protocol extensions for call control and resource status monitoring in turrets and turret switching systems |
-
2023
- 2023-02-06 CN CN202310068318.2A patent/CN116260893B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104639597A (en) * | 2013-11-08 | 2015-05-20 | 塔塔咨询服务有限公司 | System(s) and method(s) for multiple sender support in low latency fifo messaging using tcp/ip protocol |
| US10785296B1 (en) * | 2017-03-09 | 2020-09-22 | X Development Llc | Dynamic exchange of data between processing units of a system |
| CN112738190A (en) * | 2020-12-24 | 2021-04-30 | 湖南博匠信息科技有限公司 | RapidIO communication dynamic management method and system |
| CN113377091A (en) * | 2021-08-12 | 2021-09-10 | 星河动力(北京)空间科技有限公司 | Carrier rocket test launch control system |
| CN114003413A (en) * | 2021-12-30 | 2022-02-01 | 深圳佑驾创新科技有限公司 | IPC data communication method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116260893A (en) | 2023-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2005531864A (en) | System event filtering and notification to OPC clients | |
| CN112738184B (en) | Plug-in dynamic registration distributed micro-service gateway system | |
| CN113595806B (en) | Distribution network Internet of things communication architecture method based on OPCUA and MQTT protocol | |
| CN112217847A (en) | Micro service platform, implementation method thereof, electronic device and storage medium | |
| CN114866528A (en) | Data communication method based on MQTT and Websocket | |
| CN116260893B (en) | Message subscription and publishing device of data processing system | |
| CN112804109A (en) | DDS-based server client interaction model | |
| CN113630366A (en) | Internet of things equipment access method and system | |
| CN102081548A (en) | Method and system for processing Java application and SAF (service availability forum) platform | |
| CN111597157A (en) | Method for improving log processing system architecture | |
| US6496863B1 (en) | Method and system for communication in a heterogeneous network | |
| CN116405547A (en) | Message pushing method and device, processor, electronic equipment and storage medium | |
| US20040107244A1 (en) | Scalable and intelligent network platform for distributed system | |
| CN113965608A (en) | Data collection and instruction issuing system of numerical control equipment | |
| CN112787970B (en) | Method and device for subscribing event stream | |
| CN111083182B (en) | Distributed Internet of things equipment management method and device | |
| CN100379209C (en) | News processing system and method | |
| CN113347195A (en) | Dynamic mounting system and method for communication protocol packet of Internet of things equipment | |
| CN111541667A (en) | Method, equipment and storage medium for intersystem message communication | |
| US20040024732A1 (en) | Constructing a component management database for managing roles using a directed graph | |
| Ala-Hynnilä | Replacing internal communication protocol in UNIC control system | |
| CN114500660B (en) | Request processing method, device, equipment and computer readable storage medium | |
| CN114661538B (en) | Distributed database monitoring method and device, electronic equipment and storage medium | |
| CN112511528B (en) | Streaming media distribution method, system, server and storage medium | |
| CN111464520B (en) | Data processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |