CN108881369B - Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system - Google Patents
Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system Download PDFInfo
- Publication number
- CN108881369B CN108881369B CN201810373966.8A CN201810373966A CN108881369B CN 108881369 B CN108881369 B CN 108881369B CN 201810373966 A CN201810373966 A CN 201810373966A CN 108881369 B CN108881369 B CN 108881369B
- Authority
- CN
- China
- Prior art keywords
- data
- message
- node
- trigger
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/25—Routing or path finding in a switch fabric
- H04L49/252—Store and forward routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/2895—Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
Abstract
The invention relates to a data exchange method based on a data content-oriented cloud message middleware and a cloud message middleware system. The method comprises the following steps: 1) deploying nodes based on cloud message middleware at different manager stations, and forming a message transmission network by each node; 2) deploying a data content based stream trigger in each node of the message transport network; 3) and realizing data exchange across IDCs through the message transmission network, and realizing content-oriented data exchange through the stream trigger. The invention can efficiently and stably exchange cross-IDC data, and simultaneously, the invention utilizes the flow trigger based on the content to subscribe the data, and can forward the data according to the data content as required, so that consumers can independently determine the flow direction of the data and obtain the required data with fine granularity, thereby improving the quality of network transmission and reducing the network load. The multi-working mode provided by the invention can be stably used, and different application scenes can be supported.
Description
Technical Field
The invention belongs to the technical field of information, and particularly relates to a data exchange method based on a data content-oriented cloud message middleware and a cloud message middleware system.
Background
The existing message middleware adopts a publish/subscribe mode (Pub/Sub), and in an enterprise-level product, the following are mainly used: (1) IBM MQSeries: supporting deployment on a variety of platforms; providing a plurality of API interfaces; a subscription for one Topic (Topic) may be sent to multiple subscribers; complex communication protocols can be handled; reliable, one-time and only one-time (exact-once) delivery of important messages is provided. (2) And (4) SonicMQ: point-to-point and publish/subscribe modes are supported. The interface compatibility is good, the connection with other data systems can be established, the load balance is supported, and a message middleware cluster can be established; reliable, one-time and only one-time (exact-once) delivery of important messages is provided. (3) Microsoft MSMQ: supporting message routing; supporting a priority of the message; protocols supporting multiple message transmissions; message queues compatible with IBM; providing reliable, At least one (At-least-once) delivery of important messages; the authentication and encryption of the message are supported, and the security is guaranteed.
In the open source project, RabbitMQ and Apache kafka are the two most common message middleware. Wherein the RabbitMQ is message middleware implemented based on AMQP. The AMQP is an application layer standard for providing unified message service, and can achieve cross-platform and cross-programming language message communication as long as the AMQP design is followed; no special channel is needed, the topological structure is simplified, the system cost is reduced, and the performance is improved. The AMQP supports a plurality of message modes, including a request response mode (point-to-point), a publish/subscribe mode and a polling scheduling mode, and can meet a plurality of different requirements. Apache kafka is a distributed, partitionable publish/subscribe messaging system. The Producer (Producer) encapsulates the id, operation type and key of the data into a model, namely a message, and issues the model, namely the message, to a message queue of kafka, and the consumer actively reads the model from the kafka, analyzes the data referred by the model, and then directly and actively reads the required data from the Producer. Even if the message is consumed, the message cannot be deleted immediately and can be deleted after being reserved for a certain time according to the configuration requirement, so that the message is stored in the kafka in a persistent mode, the consumer is allowed to consume and take the message for multiple times, and the fault tolerance of the system is improved. For the producer, only the model needs to be published, and the complete message does not need to be uploaded, so the publishing efficiency can be high.
The above prior art has the following disadvantages:
(1) efficient cross-IDC (Internet Data Center) message exchange is not supported: the data of the existing message queue is centralized, the exchange process does not bring about physical movement and copying of the data, the message produced by a producer enters the message queue to wait for consumption, the producer and the consumer exchange the data of the 'same queue', the exchange capacity of the producer and the consumer in the same data center is very strong, but the producer and the consumer cannot be simply applied to the scene of cross-regional message exchange;
(2) content-oriented data exchange is not supported: the existing message queue data is named, the subscription of the data is required to depend on the Topic (Topic) division of a producer on the data, a consumer can only obtain the data according to the Topic but not according to the content, the consumer is highly coupled to the producer on the data content, fine-grained data screening cannot be carried out, data redundancy is easily caused, the subscription relation is difficult to dynamically expand, and the complex data exchange requirement facing the content cannot be met.
Disclosure of Invention
In order to solve the problems, the invention provides a data exchange method based on a data content-oriented cloud message middleware and a cloud message middleware system, which can efficiently and stably exchange cross-IDC data and can forward the data according to the data content as required, thereby improving the quality of network transmission and reducing the network load.
The technical scheme adopted by the invention is as follows:
a data exchange method based on cloud message middleware facing data content comprises the following steps:
1) deploying nodes based on cloud message middleware at different manager stations, and forming a message transmission network by each node;
2) deploying a data content based stream trigger in each node of the message transport network;
3) and realizing data exchange across IDCs through the message transmission network, and realizing content-oriented data exchange through the stream trigger.
Furthermore, the supervisor station comprises a sub supervisor station and a main supervisor station, wherein the sub supervisor station deploys the working nodes, and the main supervisor station deploys the main nodes; the working nodes are integrated with a message processing module Worker of the cloud message middleware, and all the working nodes are connected with each other according to a certain topology and bear the transmission function of service data; the main node integrates a message processing module Worker and a management module Master of the cloud message middleware and manages all nodes in the message transmission network; the flow trigger is deployed in the message processing module Worker.
Furthermore, the flow trigger sets the content to be triggered in a key-value pair form, binds an operation function, and processes data by using the corresponding operation function after the data flow triggers the flow trigger.
Further, when the network environment is a restricted network, the message transmission network degenerates to a transmission network with a tree structure, and the following four message forwarding modes are implemented in the transmission network with the tree structure:
a) the routing working mode is as follows: the two nodes which are not logically interconnected carry out message routing through the middle node, thereby realizing the message routing across IDCs, and simultaneously, the source node, the destination node and the middle node can audit the passing messages;
b) triggering a working mode: filtering and screening specific data through a flow trigger capable of dynamically customizing message characteristics so as to trigger customized operation;
c) and (3) a distribution working mode: when the logical superior node issues a task command or a state monitoring command with basically consistent content to the subordinate node, the task command or the state monitoring command is only sent to the subordinate node, and the subordinate node is locally copied and sent to a plurality of non-subordinate nodes in a group to reduce network load;
d) a convergence working mode: the logical subordinate nodes feed back service data or state data with basically consistent content to the superior nodes, the service data or the state data are converged to the superior nodes through routing paths, and the directly subordinate superior nodes copy and forward the service data or the state data to the non-directly subordinate superior nodes, so that the network burden is reduced.
Furthermore, when the node is used as a middle node of a passing path, whether the passing path message needs to be analyzed and processed locally or not is configured according to the requirements of the service system, and if the passing path message needs to be analyzed and processed locally, the message is transferred to the service system for processing; if not, the bypass message is audited and forwarded directly at the working node.
A cloud message middleware system facing data content comprises nodes based on cloud message middleware and deployed in different manager stations, wherein each node forms a message transmission network; deploying a stream trigger based on data content in each node of the message transmission network; the message transmission network realizes data exchange across IDCs and realizes content-oriented data exchange through the stream trigger.
Furthermore, the supervisor station comprises a sub supervisor station and a main supervisor station, wherein the sub supervisor station deploys the working nodes, and the main supervisor station deploys the main nodes; the working nodes are integrated with a message processing module Worker of the cloud message middleware, and all the working nodes are connected with each other according to a certain topology and bear the transmission function of service data; the main node integrates a message processing module Worker and a management module Master of the cloud message middleware and manages all nodes in the message transmission network; the flow trigger is deployed in the message processing module Worker.
Further, the message processing module Worker includes:
the message routing submodule: the system is responsible for carrying out message routing, auditing and pushing across IDCs among all manager stations;
a flow trigger: the trigger is a trigger capable of dynamically customizing message characteristics and is responsible for filtering and screening specific data so as to trigger specified operation;
the message distribution and aggregation sub-module: the system is responsible for realizing one-to-many data transmission by a distribution and aggregation working mode aiming at data with basically the same content when the network environment is a limited network, and reducing the network load;
a local processing submodule: and when the destination of the message flow is a local manager station, the message is processed in the working node and is delivered to the service system for processing through the local area network.
Furthermore, the flow trigger sets the content to be triggered in a key-value pair form, binds an operation function, and processes data by using the corresponding operation function after the data flow triggers the flow trigger.
Furthermore, a supervisory module hypervisor is also arranged in the working node and the main node, and is a supervisory program of the message processing module Worker, and is responsible for process daemon, memory recovery and zombie process killing so as to ensure the stability of the program process of the message processing module Worker.
The key points of the technology of the invention are as follows: a) the data transmission network is formed by cloud message middleware, and a message routing forwarding model is realized at an application layer; b) the method has the advantages that the data content-oriented stream trigger is integrated, so that a user can actively define and configure the content to be subscribed in a multi-dimension mode to obtain real-time and effective data, and meanwhile, the performance of the real-time data in cross-IDC data exchange is improved; c) following the message circulation process, ensuring the stable and effective circulation of data in the system; d) four working modes are provided under the limited network so as to meet various functional requirements.
The system provided by the invention is used for carrying out cross-IDC data interaction exchange, and has the following advantages:
1. the routing model between the nodes can completely strip data exchange from the original service system, thereby reducing the secondary development cost of the original service system;
2. the data subscription is carried out by utilizing the content-based flow trigger, so that redundant data can be effectively reduced, the network load is reduced, a consumer can independently determine the flow direction of the data, the required data can be obtained in a fine-grained manner, and the cross-IDC data exchange capacity in the topology is improved to a certain extent;
3. the invention has higher efficiency of processing data and smaller resource occupation, and can be in a relatively stable state along with the increase of concurrency and load;
4. multiple working modes can be stably used, and different application scenes can be supported.
Drawings
FIG. 1 is a general deployment architecture model diagram of the present invention.
Fig. 2 is a schematic diagram of an abstract structure of a message transmission network according to the present invention.
FIG. 3 is a diagram illustrating matching of data and triggers in the same dimension.
Fig. 4 is a schematic diagram of matching data and triggers with different dimensions, where (a) the diagram shows matching of high-dimensional data and low-dimensional triggers, and (b) the diagram shows matching of low-dimensional data and high-dimensional triggers.
Fig. 5 is a schematic diagram of the transition process of the message transition state machine in the present invention.
Fig. 6(a) -6 (d) are schematic data flow diagrams of four operation modes of the present invention, where fig. 6(a) is a routing operation mode, fig. 6(b) is a triggering operation mode, fig. 6(c) is a convergence operation mode, and fig. 6(d) is a distribution operation mode.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, the present invention shall be described in further detail with reference to the following detailed description and accompanying drawings.
The main principle of the invention is as follows: (1) a topological network with cloud message middleware as nodes is established, and an independent routing mode is designed at an application layer, so that the message can flexibly and efficiently realize cross-IDC flow; in the routing process, data is sent, subscribed, distributed and converged according to needs, so that each IDC obtains and only obtains data belonging to the IDC, redundant transmission of the data is effectively avoided, and the capacity of cross-region message exchange is enhanced; (2) and designing a stream trigger (or called as a data stream trigger and a trigger) based on data content to realize content-oriented data exchange. The trigger interface has high configurability and strong functionality, supports real-time screening processing of real-time and heterogeneous data, forms a 'subscription/push' mode, ensures that the flow direction of the data is completely determined by a consumer, a producer does not carry out theme division on the data, the consumer obtains data content through dynamic personalized configuration, the whole process is decoupled from the producer, the data screening is flexible, the granularity is small, supports complex data exchange facing to the content, and is beneficial to improving the capability of cross-IDC message exchange.
In order to meet the requirement of efficient cross-IDC exchange of open data in a distributed system, the cloud message middleware is utilized to deploy nodes on IDCs in different regions, and the data of each IDC can be efficiently routed and transmitted through the nodes in a message form, so that the data of each IDC can be efficiently cross-region exchanged; meanwhile, the cloud message middleware supports a content-oriented stream trigger, and can realize real-time subscription and real-time processing of stream data.
The cloud messaging middleware system can be directly deployed in the internet application system. The management machine station describes a machine room environment of each IDC, the cloud message middleware is used as a service system to operate on nodes of different management machine stations and are mutually interconnected, and a message transmission network with the cloud message middleware as the nodes is constructed, so that the service data transmission requirement of an original data service system of the IDC is supported.
The architecture model deployed by the invention is shown in fig. 1 and mainly comprises the following components:
1) a supervisor station: the IDC machine room environment is a general term and comprises an external network interface, a node of a cloud message middleware, a service system and the like. Each supervisor station is a local area network that is geographically separated. As shown in fig. 1, the manager station includes a sub manager station and a main manager station, where the sub manager station deploys a working node of the cloud message middleware to provide a data transmission and processing service for the data transmission network, and the main manager station deploys a master node of the cloud message middleware to have a function of managing the sub management base station in the network except for providing a data transmission and processing service for the data transmission network.
2) And (4) working nodes: the system is characterized in that a server stand-alone (or cluster) with cloud message middleware is deployed, a message processing module (Worker) integrating the cloud message middleware is connected with other working nodes of a message transmission network according to a certain topology, and as shown in fig. 2, the system bears the transmission function of service data;
3) a master node: the management node is a management node which is deployed with a cloud message middleware, integrates a message processing module (Worker) and a management module (Master) of the cloud message middleware, and can manage all nodes in a message transmission network;
4) master: the management module is a management program of the cloud message middleware, is only deployed on the main node and mainly comprises various timers (monitors) to realize active monitoring and management on each working node;
5) and (2) Worker: the message processing module is a core function module of the cloud message middleware and is used for processing messages, including the shunting of different types of messages, the routing of the messages, a stream trigger for subscribing the messages and the like;
6) supervisor: the monitoring module is a monitoring program of a message processing module (Worker) and is responsible for functions of process daemon, memory recovery, zombie process killing and the like so as to ensure the stability of the Worker program process.
7) A service system: the management machine station is originally deployed with a third-party service application system, and hopes to perform service data transmission with service systems deployed in other management machine stations through a message transmission network.
Among the above components, the working node is the main body of the present invention, and is also the core component for constructing the message transmission network, and the message processing module (Worker) is used to complete the cross-IDC data exchange facing the content. The main functional modules (submodules) forming the Worker include the following modules:
a) the message routing submodule: through the working node, message routing, auditing and pushing across IDCs can be carried out among all manager stations.
b) A flow trigger: the trigger is a trigger capable of dynamically customizing message characteristics and is used for filtering and screening specific data so as to trigger specified operation;
c) the message distribution and aggregation sub-module: when the network environment is a limited network, the message transmission network is degenerated into a tree-shaped topological structure network, and in the tree-shaped topological structure network, data with basically the same content can be transmitted in a one-to-many mode through a distribution and aggregation working mode, so that the network burden is reduced.
d) A local processing submodule: when the end point of the message flow is a local manager station, the message is processed in the working node and is delivered to the service system for processing through the local area network.
When a user configures the trigger, the stream trigger of the present invention may set the content to be triggered in the form of a key-value pair, and bind the trigger to an operation function (such as sending, discarding, counting, or other complex calculations).
However, data and triggers may be heterogeneous in content, so data and triggers may have dimensional differences. Numerical data and triggers in three-dimensional space are used here as an example. As shown in fig. 3, when the data and the trigger have the same dimension, the matching triggering can be directly performed, and once the matching is successful (the overlapped part in fig. 3 is the data with which the matching is successful), the triggered operation can be performed. In fig. 3X represents a point in a high dimensional space. But in most cases the data and the triggers are of different dimensions. As shown in fig. 4, when the dimension of the data is higher than that of the trigger, the data may be projection-reduced to the same dimension as the trigger, and then the data and the trigger with the same dimension are matched with each other; when the dimension of a trigger is higher than the dimension of the data, the data is considered to have a missing amount of information with respect to the trigger, so this trigger is ignored, i.e., no matching is performed. In the diagram (a) of fig. 4, high-dimensional data is projected to a low-dimensional trigger, and the overlapped part is data successfully matched; (b) in the figure, low-dimensional data is expanded to a high-dimensional trigger, the data (set) is expanded to virtual high-dimensional data (set), and even if the dotted line (dotted cylinder) overlaps with the trigger or the data is wrapped by the high-dimensional trigger, the dotted line (dotted cylinder) representing the data (set) is not triggered, namely, the matching is considered to be failed.
The process of message circulation in the present invention is shown in fig. 5, and includes: the message is generated (Created) by a data producer and sent to a corresponding node (Received), the content of the message is shunted (Diverted) and analyzed in the node, so that triggering (Triggered) and route forwarding (Routed) of the message are processed, and the message routing submodule analyzes the destination of the message, forwards the message to the next node (Received) and carries out similar processing. When the message reaches the destination, the message is processed locally (Landed), and the data of the message is stored and Delivered to a service system and an analysis system (Delivered). "redirect" in fig. 5 refers to determining the destination of message transmission, and "match" refers to matching data to triggers.
When the network environment is a limited network (the communication between the nodes is limited and data exchange cannot be directly carried out), the invention can adopt a master-slave structure deployment to complete the data routing under the limited network, and at the moment, the message transmission network is degenerated into a network with a tree structure. Fig. 6(a) to fig. 6(d) show four message forwarding modes under the tree-like transmission network according to the present invention:
a) point-to-point routing (P2P), i.e., routing mode of operation, as shown in fig. 6 (a): two nodes which are not logically interconnected can carry out message routing through the middle node, thereby realizing the message routing across IDCs, and simultaneously, the source node, the destination node and the middle node can audit the passing messages;
b) the subscription triggers a specific message (subscribe/push), i.e. triggers the working mode, as shown in fig. 6 (b): and filtering and screening specific data through a data flow trigger capable of dynamically customizing the message characteristics, thereby triggering the customized operation.
c) One-to-many distribution, i.e., distribution mode of operation, as shown in fig. 6 (d): when a logic superior node (such as a master node) issues a task command or a state monitoring command with basically consistent content to a subordinate node, the task command or the state monitoring command can be only sent to the subordinate node, and the subordinate node is locally copied and sent to a plurality of non-subordinate nodes in a group manner, so that the network load is reduced;
d) many-to-one convergence, i.e. a convergence working mode, as shown in fig. 6 (c): the service data or the state data with basically consistent content are fed back to the upper node by the logic lower node, can be gathered to the upper node through the routing path, and are copied and forwarded to the non-directly-subordinate upper node by the directly-subordinate upper node, so that the network burden is reduced.
In order to ensure efficient routing and safety audit to a certain extent, when a node is used as a middle node of a passing route, whether the passing message is analyzed and processed locally or not can be configured according to the requirements of a service system, and if the service processing is needed, the message needs to be transferred to the service system for processing locally; if not, the passing-by message is directly audited and forwarded at the working node without being processed by other modules of the supervisor station.
Example 1 takes the data subscription function of the invention as an example:
1. a consumer sets a stream trigger to subscribe the content of the character string data;
2. a producer sets 1 client, produces 100 pieces of character string data with the size of 590B in total and delivers the character string data to the node of the invention;
3. if the data triggers the trigger, pushing the data to a node where the consumer is located, and if the data does not trigger the trigger, discarding the data;
4. and recording the average time delay of the message completing the life cycle and the memory resources occupied by the message.
Table 1 shows the experimental results of example 1.
TABLE 1 Experimental results for example 1
| Examples of the invention | Message type | Message size | Degree of concurrency | Amount of test data | Average time delay | Memory usage |
| 1 | Character string | 590B | 1 | 100 ten thousand | 21ms | 50.1M |
Example 2 takes the data subscription function of the invention as an example:
1. a consumer sets a stream trigger to subscribe the content of the character string data;
2. a producer sets 20 clients, produces 100 pieces of character string data with the size of 590B in total and delivers the data to the node of the invention;
3. if the data triggers the trigger, pushing the data to a node where the consumer is located, and if the data does not trigger the trigger, discarding the data;
4. and recording the average time delay of the message completing the life cycle and the memory resources occupied by the message.
Table 2 shows the experimental results of example 2.
TABLE 2 Experimental results for example 2
| Examples of the invention | Message type | Message size | Degree of concurrency | Amount of test data | Average time delay | Memory usage |
| 2 | Character string | 590B | 20 | 100 ten thousand | 38ms | 50.3M |
Example 3 takes the data subscription function of the invention as an example:
1. a consumer sets a stream trigger to subscribe the content of the file data;
2. the producer sets 1 client, produces 100 ten thousand pieces of file data with the size of 79KB and delivers the file data to the node of the invention;
3. if the data triggers the trigger, pushing the data to a node where the consumer is located, and if the data does not trigger the trigger, discarding the data;
4. and recording the average time delay of the message completing the life cycle and the memory resources occupied by the message.
Table 3 shows the experimental results of example 3.
TABLE 3 Experimental results for example 3
| Examples of the invention | Message type | Message size | Degree of concurrency | Amount of test data | Average time delay | Memory usage |
| 3 | Document | 79KB | 1 | 100 ten thousand | 45ms | 50.35M |
Example 4 takes the data subscription function of the invention as an example:
1. a consumer sets a stream trigger to subscribe the content of the file data;
2. the producer sets 20 clients, produces 100 ten thousand pieces of file data with the size of 79KB and delivers the file data to the node of the invention;
3. if the data triggers the trigger, pushing the data to a node where the consumer is located, and if the data does not trigger the trigger, discarding the data;
4. and recording the average time delay of the message completing the life cycle and the memory resources occupied by the message.
Table 4 shows the experimental results of example 4.
TABLE 4 Experimental results for EXAMPLE 4
| Examples of the invention | Message type | Message size | Degree of concurrency | Amount of test data | Average time delay | Memory usage |
| 4 | Document | 79KB | 20 | 100 ten thousand | 60ms | 50.35M |
According to the experimental results of the four examples, when data are transmitted under a large load and a large concurrency degree, redundant data can be effectively reduced after messages are screened by using the data flow trigger, the cloud message middleware is guaranteed not to generate congestion of the messages along with accumulation of data quantity, the message processing module can well recover the memory in the working process, the problems of memory leakage and the like are avoided, and the cloud message middleware is guaranteed to stably and durably operate.
The above embodiments are only intended to illustrate the technical solution of the present invention and not to limit the same, and a person skilled in the art can modify the technical solution of the present invention or substitute the same without departing from the spirit and scope of the present invention, and the scope of the present invention should be determined by the claims.
Claims (7)
1. A data exchange method based on cloud message middleware facing data content is characterized by comprising the following steps:
1) deploying nodes based on cloud message middleware at different manager stations, and forming a message transmission network by each node;
2) deploying a data content based stream trigger in each node of the message transport network; the stream trigger sets the content to be triggered in a key-value pair mode, binds an operation function, and processes data by using the corresponding operation function after the stream trigger is triggered by the data stream; when the data trigger and the stream trigger have the same dimensionality, directly performing matching triggering; when the dimensionality of the data is higher than the dimensionality of the flow trigger, projecting the data dimensionality reduction to the dimensionality same as that of the flow trigger, and then matching the data and the flow trigger with the same dimensionality; when the dimension of the flow trigger is higher than that of the data, the data is considered to have information quantity missing relative to the flow trigger, and the flow trigger is ignored, namely, no matching is carried out;
3) realizing data exchange across IDCs through the message transmission network, and realizing content-oriented data exchange through the stream trigger;
when the network environment is a restricted network, the message transmission network degenerates to a transmission network with a tree structure, and the following four message forwarding modes are implemented in the transmission network with the tree structure:
a) the routing working mode is as follows: the two nodes which are not logically interconnected carry out message routing through the middle node, thereby realizing the message routing across IDCs, and simultaneously, the source node, the destination node and the middle node can audit the passing messages;
b) triggering a working mode: filtering and screening specific data through a flow trigger capable of dynamically customizing message characteristics so as to trigger customized operation;
c) and (3) a distribution working mode: when the logical superior node issues a task command or a state monitoring command with basically consistent content to the subordinate node, the task command or the state monitoring command is only sent to the subordinate node, and the subordinate node is locally copied and sent to a plurality of non-subordinate nodes in a group to reduce network load;
d) a convergence working mode: the logical subordinate nodes feed back service data or state data with basically consistent content to the superior nodes, the service data or the state data are converged to the superior nodes through routing paths, and the directly subordinate superior nodes copy and forward the service data or the state data to the non-directly subordinate superior nodes, so that the network burden is reduced.
2. The method of claim 1, wherein the supervisor stations include child supervisor stations that deploy worker nodes and a master supervisor station that deploys a master node; the working nodes are integrated with a message processing module Worker of the cloud message middleware, and all the working nodes are connected with each other according to a certain topology and bear the transmission function of service data; the main node integrates a message processing module Worker and a management module Master of the cloud message middleware and manages all nodes in the message transmission network; the flow trigger is deployed in the message processing module Worker.
3. The method according to claim 1, characterized in that when the node is used as a middle node of the path, whether the path message needs to be analyzed and processed locally or not is configured according to the requirement of the service system, if so, the message is transferred to the service system for processing; if not, the bypass message is audited and forwarded directly at the working node.
4. A data content oriented cloud messaging middleware system employing the method of claim 1, comprising cloud messaging middleware based nodes deployed at different supervisor stations, each of said nodes constituting a message transport network; deploying a stream trigger based on data content in each node of the message transmission network; the message transmission network realizes data exchange across IDCs and realizes content-oriented data exchange through the stream trigger.
5. The cloud message middleware system of claim 4 wherein the manager stations include child manager stations that deploy work nodes and master manager stations that deploy master nodes; the working nodes are integrated with a message processing module Worker of the cloud message middleware, and all the working nodes are connected with each other according to a certain topology and bear the transmission function of service data; the main node integrates a message processing module Worker and a management module Master of the cloud message middleware and manages all nodes in the message transmission network; the flow trigger is deployed in the message processing module Worker.
6. The cloud messaging middleware system of claim 5, wherein the message processing module Worker comprises:
the message routing submodule: the system is responsible for carrying out message routing, auditing and pushing across IDCs among all manager stations;
a flow trigger: the trigger is a trigger capable of dynamically customizing message characteristics and is responsible for filtering and screening specific data so as to trigger specified operation;
the message distribution and aggregation sub-module: the system is responsible for realizing one-to-many data transmission by a distribution and aggregation working mode aiming at data with basically the same content when the network environment is a limited network, and reducing the network load;
a local processing submodule: and when the destination of the message flow is a local manager station, the message is processed in the working node and is delivered to the service system for processing through the local area network.
7. The cloud message middleware system of claim 5, wherein the working node and the main node are further provided with a supervisory module Supervisor, which is a supervisory program of the message processing module Worker and is responsible for process daemon, memory recovery and zombie process killing so as to ensure stability of the program process of the message processing module Worker.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810373966.8A CN108881369B (en) | 2018-04-24 | 2018-04-24 | Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810373966.8A CN108881369B (en) | 2018-04-24 | 2018-04-24 | Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108881369A CN108881369A (en) | 2018-11-23 |
| CN108881369B true CN108881369B (en) | 2020-09-29 |
Family
ID=64326394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810373966.8A Active CN108881369B (en) | 2018-04-24 | 2018-04-24 | Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108881369B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109784090A (en) * | 2018-12-27 | 2019-05-21 | 浪潮软件股份有限公司 | A kind of method and system for realizing multi-tenant control based on cloud messaging service |
| CN111597058B (en) * | 2020-04-17 | 2023-10-17 | 微梦创科网络科技（中国）有限公司 | Data stream processing method and system |
| CN112235398A (en) * | 2020-10-12 | 2021-01-15 | 南威软件股份有限公司 | Data transmission method supporting large data volume |
| CN112688998B (en) * | 2020-12-17 | 2023-03-14 | 中国航空工业集团公司成都飞机设计研究所 | Main data subscription pushing method capable of configuring with permission |
| CN113259195B (en) * | 2021-05-10 | 2022-07-05 | 中国人民解放军63660部队 | Message middleware performance test method |
| CN113342552A (en) * | 2021-07-05 | 2021-09-03 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN113489792B (en) * | 2021-07-07 | 2023-02-03 | 上交所技术有限责任公司 | Method for reducing network transmission times among data centers in cross-data center cluster consensus algorithm |
| CN114327406B (en) * | 2021-12-30 | 2022-08-12 | 重庆允成互联网科技有限公司 | Trigger data filtering method, trigger configuration method and computer storage medium |
| CN115150465A (en) * | 2022-07-08 | 2022-10-04 | 中国建设银行股份有限公司 | Information processing method, device, equipment, storage medium and product |
| CN115086419A (en) * | 2022-07-24 | 2022-09-20 | 浙江大学 | Message mechanism-based cross-three-party system message transmission center system |
| CN115720236B (en) * | 2022-11-18 | 2024-04-19 | 北京航天发射技术研究所 | Lightweight communication middleware based on heterogeneous network |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1538316A (en) * | 2003-04-17 | 2004-10-20 | 联想(北京)有限公司 | Message intermediate item system possessing level topological structure and message transmission method |
| US7093261B1 (en) * | 2000-07-28 | 2006-08-15 | Fair Isaac Corporation | Message integration framework for multi-application systems |
| CN101976210A (en) * | 2010-08-26 | 2011-02-16 | 中国石油集团川庆钻探工程有限公司 | Well site data remote transmission method based on message middleware |
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN104468831A (en) * | 2014-12-26 | 2015-03-25 | 山东中创软件商用中间件股份有限公司 | Message service network cluster based on network topology |
| CN105119761A (en) * | 2015-09-18 | 2015-12-02 | 成都视达科信息技术有限公司 | Operation and maintenance monitoring and solution integrated management system and method |
| CN106126349A (en) * | 2016-06-14 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of remote invocation method, message-oriented middleware cluster and system |
| CN106230922A (en) * | 2016-07-28 | 2016-12-14 | 上海斐讯数据通信技术有限公司 | A kind of news subscribing method, message production unit and message reception units |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102298624B (en) * | 2011-08-15 | 2014-08-20 | 东北大学 | Approximate dynamic ring matching method for homogeneous and symmetric publication and subscription system |
| CN105933444B (en) * | 2016-06-27 | 2017-06-09 | 焦点科技股份有限公司 | Based on the service discovery method that registration center and caching mechanism are cooperateed with |
| US10560407B2 (en) * | 2016-10-06 | 2020-02-11 | Sap Se | Payload description for computer messaging |
-
2018
- 2018-04-24 CN CN201810373966.8A patent/CN108881369B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7093261B1 (en) * | 2000-07-28 | 2006-08-15 | Fair Isaac Corporation | Message integration framework for multi-application systems |
| CN1538316A (en) * | 2003-04-17 | 2004-10-20 | 联想(北京)有限公司 | Message intermediate item system possessing level topological structure and message transmission method |
| CN101976210A (en) * | 2010-08-26 | 2011-02-16 | 中国石油集团川庆钻探工程有限公司 | Well site data remote transmission method based on message middleware |
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN104468831A (en) * | 2014-12-26 | 2015-03-25 | 山东中创软件商用中间件股份有限公司 | Message service network cluster based on network topology |
| CN105119761A (en) * | 2015-09-18 | 2015-12-02 | 成都视达科信息技术有限公司 | Operation and maintenance monitoring and solution integrated management system and method |
| CN106126349A (en) * | 2016-06-14 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of remote invocation method, message-oriented middleware cluster and system |
| CN106230922A (en) * | 2016-07-28 | 2016-12-14 | 上海斐讯数据通信技术有限公司 | A kind of news subscribing method, message production unit and message reception units |
Non-Patent Citations (2)
| Title |
|---|
| 《基于内容的订阅方法及其在MQSeries中的实现》;刘振涛，熊璋，欧阳元新;《计算机工程》;20040831(第16期);第91-93页 * |
| 《树型通信网络中消息中间件的研究与设计》;于源猛;《中国优秀硕士学位论文全文数据库 信息科技辑》;20101215;I136-83 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108881369A (en) | 2018-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108881369B (en) | Data exchange method based on data content-oriented cloud message middleware and cloud message middleware system | |
| CN107590182B (en) | Distributed log collection method | |
| Agarwal et al. | The Totem multiple-ring ordering and topology maintenance protocol | |
| US7177917B2 (en) | Scaleable message system | |
| US8719780B2 (en) | Application server with a protocol-neutral programming model for developing telecommunications-based applications | |
| US20140280988A1 (en) | System and method for parallel multiplexing between servers in a cluster | |
| US9525566B2 (en) | Self-managed mediated information flow | |
| CN106789755A (en) | Inter-network data interchange platform | |
| EP2838243B1 (en) | Capability aggregation and exposure method and system | |
| CN105337780B (en) | A kind of server node configuration method and physical node | |
| CN110955534A (en) | Middleware-based business process automation system and method | |
| CN103581307A (en) | Publishing/subscribing system based on clusters and method for guaranteeing reliability of publishing/subscribing system based on clusters | |
| WO2022120806A1 (en) | Multi-cloud distributed messaging method and system for high performance computing | |
| Tiwari et al. | Reconfigurable streaming for the mobile edge | |
| CN106648801A (en) | Automatic node adding and deleting method for Hadoop cluster | |
| CN110366056B (en) | Method, device, equipment and storage medium for realizing ASON business model | |
| Kong et al. | Run-time recovery and failure analysis of time-triggered traffic in time sensitive networks | |
| CN116489214A (en) | Unified service scheduling method, device, medium and equipment based on micro service gateway | |
| Rodrigues et al. | Topology-aware algorithms for large-scale communication | |
| CN113923116A (en) | Method for managing multiple wide area network clusters based on one platform | |
| Almeida et al. | Using light-weight groups to handle timing failures in quasi-synchronous systems | |
| US20170310551A1 (en) | Modular controller in software-defined networking environment and operating method thereof | |
| CN114745377A (en) | Edge cloud cluster service system and implementation method | |
| CN109257435B (en) | Communication system of multi-instance multi-subsystem application based on message middleware | |
| CN106789380A (en) | A kind of virtual machine network integration supervisory systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |