CN113542314B - Method and system for realizing subscription mode based on keyword filtering - Google Patents
Method and system for realizing subscription mode based on keyword filtering Download PDFInfo
- Publication number
- CN113542314B CN113542314B CN202010285711.3A CN202010285711A CN113542314B CN 113542314 B CN113542314 B CN 113542314B CN 202010285711 A CN202010285711 A CN 202010285711A CN 113542314 B CN113542314 B CN 113542314B
- Authority
- CN
- China
- Prior art keywords
- message
- subscriber
- keyword
- filtering
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9536—Search customisation based on social or collaborative filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method and a system for realizing a subscription mode based on keyword filtering, and relates to the technical field of network communication. The method comprises the following steps: after receiving a message of a specified message topic issued by a publisher, a message broker stores the received message into the tail of one or more queues corresponding to the message topic, judges whether to filter keywords according to the identification prefix of the message topic, and if so, stores the message into a container capable of being persisted for filtering; when the message agent receives a request of pulling a message from a subscriber and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber. The invention can filter the repeated and invalid messages, so that the subscriber only receives the valid messages and the receiving reliability of the messages is ensured.
Description
Technical Field
The invention relates to the technical field of network communication, in particular to a subscription mode realization method and a subscription mode realization system based on keyword filtering.
Background
In the management process of the telecommunication network, a system has massive messages to carry out network transmission in a message subscription mode, which relates to the problem of message effectiveness. Because a large amount of messages are transmitted to the network, great pressure is brought to the processing capacity and the storage capacity of the network, and invalid messages definitely waste the network bandwidth, occupy the processing resources of the valid messages and consume the storage space. Especially, under the condition that the subscriber cannot judge to receive the message baseline, all historical messages corresponding to the message topic need to be received once, and the final effective message can be obtained.
Therefore, how to filter the repeated and invalid messages so that the subscriber only receives the valid messages and the receiving reliability of the messages is ensured is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The present invention aims to overcome the defects of the background art, and provides a method and a system for implementing a subscription mode based on keyword filtering, which can filter repeated and invalid messages, enable subscribers to receive only valid messages, and ensure the reliability of message reception.
In order to achieve the above object, the present invention provides a method for implementing a subscription mode based on keyword filtering, comprising the following steps:
A. the message broker receives a message of a specified message topic issued by a publisher; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
B. the message agent stores the received message into the tail part of one or more queues corresponding to the message subject; judging whether to filter keywords according to the identification prefix of the message theme, if so, transferring the message to a durable container for filtering;
C. when the message agent receives a request of pulling a message from a subscriber and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber.
On the basis of the above technical solution, in the step B, whether to perform keyword filtering is determined according to the identification prefix of the message topic, and if yes, the message is transferred to a durable container for filtering, which specifically includes the following operations:
judging whether to filter keywords according to the identification prefix of the message theme, if so, continuously judging whether information of the message corresponding to the keywords is stored in the persistence container, if so, deleting the information and then inserting the information of the message into the persistence container, otherwise, directly inserting the information of the message into the persistence container; the information of the message includes: a key of the message, the complete content of the message, an identification number of the queue in which the message is located, an offset in the message re-queue, and a timestamp of the message.
On the basis of the above technical solution, in the step C, according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber, the filtered message is pulled and returned to the subscriber in the persistence container and/or the corresponding queue, and the specific steps include:
1) firstly, according to the message receiving base line of the subscriber, determining the position of the message to be pulled:
if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue;
if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled from the corresponding topic in the persistence container, and returning a mark pulled from the queue to the subscriber after judging that the data is completely taken from the persistence container according to the request condition of the subscriber; then pulling from the last message in the corresponding queue;
if the message receiving baseline of the subscriber is the specified baseline of the subscriber, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
2) and then according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled:
if the subscriber does not give the keyword, pulling all the messages at the pulling position and the subsequent positions;
if the subscriber gives the specified keyword, the message which contains the specified keyword and is at the pulling position and then is pulled.
On the basis of the technical scheme, the method further comprises the following operations: after receiving the message returned by the message agent, the subscriber takes the message with the latest timestamp as the final message for the message with the same keyword and different queues, and transmits the final message to the upper application.
On the basis of the above technical solution, when the message broker is set to be more than one, and one of the message brokers is used as a master node, and the remaining message brokers are used as slave nodes, the method further includes the following operations:
when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; after receiving the new message sent by the publisher, the subsequent master node forwards the new message to the slave node, and the slave node processes the new message according to the operation in the step B; when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message according to the operation of step A, B, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message according to the operation of step C.
The invention also provides a system for realizing the subscription mode based on the keyword filtering, which comprises a publisher, a message agency and a subscriber; the message agent comprises a message receiving module, a message subject identification module, a message processing module and a message pulling module;
the message receiving module is configured to: receiving a message which is issued by a publisher and specifies a message subject; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
the message topic identification module is configured to: judging whether to filter keywords according to the identification prefix of the message theme;
the message processing module is configured to: storing the received messages into the tail parts of one or more queues corresponding to the message subjects; when the keyword filtering is needed, the message is transferred to a durable container for filtering;
the message pulling module is configured to: when a request of pulling a message by a subscriber is received and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence-capable container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber.
On the basis of the technical scheme, the message processing module stores the message in a persistent container for filtering, and the method specifically comprises the following operations: when the keyword filtering is determined to be needed, transferring the information of the message to a persistence container, wherein the information of the message comprises: the method comprises the steps of generating a message queue, wherein the message queue comprises key words of the message, complete content of the message, an identification number of the queue where the message is located, offset in the message queue and a timestamp of the message; before storing, checking whether the information of the message corresponding to the keyword exists, and if so, covering the original message with a new message.
On the basis of the technical scheme, the message pulling module pulls the filtered message to return to the subscriber in the persistence container and/or the corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber, and the specific operations include:
firstly, according to the message receiving base line of the subscriber, determining the position of the message to be pulled: if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue; if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled from the corresponding topic in the persistence container, and returning a mark pulled from the queue to the subscriber after judging that the data is completely taken from the persistence container according to the request condition of the subscriber; then pulling from the last message in the corresponding queue; if the message receiving baseline of the subscriber is the subscriber specified baseline, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
and then according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled: if the subscriber does not give the keyword, pulling all the messages at the pulling position and the subsequent positions; if the subscriber gives the specified keyword, the message which contains the specified keyword and is at the pulling position and then is pulled.
On the basis of the above technical solution, the subscriber is provided with a message filtering module for: and after receiving the message returned by the message agent, taking the message with the latest timestamp as a final message for the message with the same keyword and different queues, and transmitting the final message to the upper application.
On the basis of the technical scheme, the system comprises more than one message agent, wherein one message agent is used as a main node, and the other message agents are used as slave nodes;
when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; the subsequent main node receives the new message sent by the publisher and forwards the new message to the slave node, and the slave node processes the message by using the message subject identification module and the message processing module;
when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message by using the message receiving module, the message subject identification module and the message processing module, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message by using the message pulling module.
The invention has the beneficial effects that:
(1) in the invention, when the message agent processes the message sent by the publisher, the message topic is the topic of keyword filtering, and the filtering persistence is directly carried out in the container capable of persistence, namely for the messages with the same keywords, the new message is covered on the old message, so that the latest message can be always stored in the container capable of persistence, thereby achieving the effect of filtering the latest message; meanwhile, the messages of all message themes are stored in a queue tail insertion mode, so that the last message of the keywords in each queue is the latest message, and the effect of compressing and filtering historical messages is achieved.
Meanwhile, when the message agent processes the request of pulling the message from the subscriber, the message agent will pull the message back to the subscriber in the persistent container and/or the corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber. The message pulling process is based on the message receiving baseline of the subscriber and the message keywords given by the subscriber to pull the targeted filtering message, so that the purpose of avoiding the subscriber from passively receiving a large amount of useless messages can be achieved, and the effectiveness and reliability of message receiving are ensured.
(2) In the invention, a filtering mechanism is added at the subscriber end, so that the subscriber end can combine messages from a plurality of queues, and takes the message with the latest timestamp as the final message for the messages with the same keyword and different queues, and transmits the final message to the upper application, thereby realizing the secondary filtering of the subscriber end and further ensuring the effectiveness of message acceptance.
(3) The invention designs the master and slave working modes of the message agent aiming at the condition that the message agent can not work normally due to faults, thereby effectively avoiding the influence on the normal use of the subscriber end due to the faults of the message agent and further ensuring the reliability of the subscription mode.
(4) The invention provides a simple and efficient keyword filtering scheme with strong functions, and users can apply the functions only by simply appointing the message subject names between publishers and subscribers without paying attention to the implementation principle of the keyword filtering scheme.
Drawings
FIG. 1 is a schematic diagram illustrating a method for implementing a subscription model based on keyword filtering according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for implementing a subscription model based on keyword filtering according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating step B according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a subscription mode implementation method based on keyword filtering according to a second embodiment of the present invention;
fig. 5 is a block diagram of a subscription mode implementation system based on keyword filtering according to a third embodiment of the present invention.
Detailed Description
First, the design principle of the present invention is further explained: in a traditional message subscription mode provided by a message broker, a subscriber cannot avoid receiving messages with repeated keywords when subscribing the messages. Such messages are time-sensitive in many scenarios, and the subscriber only needs to accept the last message. For example, in the case of a warning lamp in a network management system, the user interface only needs to acquire the last warning lamp status of each network element, and in such a scenario, it is not necessary to accept all messages.
Therefore, in order to avoid the subscriber accepting the repeatedly invalid message and ensure the validity and reliability of the message acceptance, as shown in fig. 1, the main solution proposed by the present invention is: (1) after receiving the message sent by the message publisher, the message broker firstly appends the message to the tail of the corresponding queue, then judges whether the message theme corresponding to the queue needs to be subjected to keyword filtering processing, if so, the message is transferred to a persistence container, and filtering persistence can be carried out in the persistence container; specifically, for messages with the same keyword, a new message is overlaid on an old message, so that the latest message can be stored in the persistent container all the time, and the effect of filtering out the latest message is achieved. (2) After receiving a request of pulling the message from the subscriber, the message agent pulls the filtered message back to the subscriber in a persistent container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber; the message pulling process is based on the message receiving baseline of the subscriber and the message keywords given by the subscriber to pull the targeted filtering message, so that the purpose of avoiding the subscriber from passively receiving a large amount of useless messages can be achieved, and the effectiveness and reliability of message receiving are ensured.
In order that those skilled in the art will better understand the present invention, the following detailed description of the invention is provided in conjunction with the accompanying drawings and the detailed description of the invention.
However, it should be noted that: the examples to be described next are only some specific examples, and are not intended to limit the embodiments of the present invention necessarily to the following specific steps, values, conditions, data, orders, and the like. Those skilled in the art can, upon reading this specification, utilize the concepts of the present invention to construct more embodiments than those specifically described herein.
Example one
Referring to fig. 2, the present embodiment provides a method for implementing a subscription model based on keyword filtering, where the method includes the following steps:
A. the message broker receives a message of a specified message topic issued by a publisher; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
B. the message agent stores the received message into the tail part of one or more queues corresponding to the message subject; judging whether to filter keywords according to the identification prefix of the message theme, if so, transferring the message to a durable container for filtering; it will be appreciated that a message topic may have one or more queues, the number of queues may be specified in a configuration file or in constructing a message topic, and typically a message topic may default to one queue.
C. When the message agent receives a request of pulling a message from a subscriber and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber.
It can be understood that, when a publisher sends a message to a specified message topic, the publisher adds an identification prefix to the name of the message topic that needs to support keyword filtering, for example: the prefix "map _" specifically adopts an identification prefix that can be agreed between a publisher, a message broker and a subscriber, and the embodiment is not specifically limited; in addition, the publisher can also carry the keywords of the current message in the message body of the sent message, and during specific operation, the keywords of the current message can be carried in the publicly identifiable part in the message body
Further, in practical operation, since there may be a plurality of queues under one message topic, as a preferred embodiment, when a publisher sends a message to a queue of a message topic specified in a message broker, it may send messages with the same keyword to the same queue as much as possible. Because the filtering of the message broker is directed to the filtering of the messages in the queue (i.e. pulling the last message of the key word on the corresponding queue), the filtering effect can be better achieved only if the messages with the same key word are in the same queue as much as possible.
Further, referring to fig. 3, in practical operation, step B of the method specifically includes the following operations:
b1, the message agent stores the received message into the tail of the corresponding queue;
b2, judging whether to filter keywords according to the identification prefix of the message theme, if so, turning to the step B3; otherwise, the step B is finished after a result is returned to the publisher;
b3, judging whether the information of the message corresponding to the keyword is stored in the container capable of being persisted, if so, deleting the information and then transferring to the step B4; otherwise, directly proceeding to step B4;
b4, transferring the information of the message to a persistence-able container, wherein the information of the message comprises: keywords of the message, complete content of the message, an identification number of a queue where the message is located, an offset in the message re-queue, a timestamp of the message and the like; and, after returning the result to the publisher, ending step B.
It can be understood from the specific operation in the step B, that in this embodiment, when the message broker processes the message sent by the publisher, as for the message topic that is the topic of keyword filtering, filtering persistence is directly performed in the container capable of persisting — for the message with the same keyword, a new message is overlaid on an old message, so that the latest message is always stored in the container capable of persisting, thereby achieving the effect of filtering the latest message; meanwhile, the messages of all message themes are stored in a queue tail insertion mode, so that the last message of the keywords in each queue is the latest message, and the effect of compressing and filtering historical messages is achieved.
Further, in actual operation, in step C of the method, according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber, the filtered message is pulled and returned to the subscriber in the persistence-capable container and/or the corresponding queue, specifically including the following steps:
first, based on the subscriber's accepted message baseline, the location of the message to be pulled is determined (i.e., where to start pulling the message is determined): (1) if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue; (2) if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled down from the corresponding topic in the persistence container, judging that the data is completely taken from the persistence container according to the request condition of the subscriber, and returning a mark pulled from the queue to the subscriber; then pulling from the last message in the corresponding queue; (3) if the message receiving baseline of the subscriber is the specified baseline of the subscriber, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
and then, according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled (i.e. determining the number of the messages to be pulled): (1) if the subscriber gives no keyword (that is, the number of keywords given by the subscriber is 0, which means that the message with any value of the keyword can be accepted), pulling all the messages at the pulling position and after; (2) if the subscriber gives a specified keyword (that is, the number of keywords given by the subscriber is not 0, which means that only the message of the specified keyword is accepted), the message including the specified keyword at the above-mentioned pull position and thereafter is pulled.
To facilitate understanding of the above operations, the following detailed description is given with specific examples:
example 1, assume that a message acceptance baseline of a subscriber is collected from the current time, and the subscriber does not give a keyword of a message to be pulled; the message broker will pull the last message of all keys in each queue.
Example 2, assume that the message receiving baseline of a certain subscriber is from the current time, and the subscriber gives a keyword of a message to be pulled as a; the message broker will pull the last message of key a in each queue.
Example 3, assume that the message receiving baseline of a certain subscriber is uncertain, and the subscriber does not give the keywords of the message to be pulled; the message broker will pull the latest one of all keys from the persistency container first and then start with the last message of all keys in each queue.
Example 4, assume that the message receiving baseline of a subscriber is uncertain, and the subscriber gives a keyword of a message to be pulled as B; the message broker will pull the latest message for key B from the persistency container first and then start with the last message for key B in each queue.
Example 5, assume that the accepted message baseline of a subscriber is the specified baseline, and the subscriber does not give the keyword of the message to be pulled; the message broker will pull all the keys to the specified positions in the queues and the messages thereafter.
Example 6, assume that the message-receiving baseline of a subscriber is the specified baseline, and the subscriber gives the keyword C for the message to be pulled; the message broker will pull the message with the key C specified in the respective queue and thereafter.
It can be understood from the above specific operation of step C, that in this embodiment, when processing a request of pulling a message from a subscriber, the message broker returns the pulled message to the subscriber in the persistable container and/or the corresponding queue according to the baseline condition of the message received by the subscriber and the keyword condition of the message given by the subscriber. The message pulling process is targeted pulling, the latest message is ensured to be pulled as much as possible, and even if the historical message is pulled, the last message in the queue is pulled, so that the historical message is compressed and filtered, the aim of avoiding a subscriber from passively receiving a large amount of useless messages is finally fulfilled, and the effectiveness and the reliability of message receiving are ensured.
Further, as a preferred implementation manner in this embodiment, the following operations are further included after step C: after receiving the message returned by the message agent, the subscriber takes the message with the latest timestamp as the final message for the message with the same keyword and different queues, and transmits the final message to the upper application.
It can be understood that, since the existence of multiple queues can be supported in one message topic, and the filtering is a function inside a single queue, if multiple queues exist, a subscriber may acquire a message from multiple queues for the same keyword, and there may be a case where the message between the queues is duplicated or invalid. In order to solve this problem, in the preferred embodiment, a filtering mechanism is added to the subscriber end, so that the subscriber end can merge messages from multiple queues, and for messages with the same keyword but different queues, take the message with the latest timestamp as the final message, and transmit the final message to the upper layer application, thereby implementing re-filtering. More specifically, in practical applications, the subscriber end may maintain the received message keyword and the MAP of the message timestamp when starting to pull the message, and then instruct the subscriber end to receive the messages according to the sequence of the message timestamps. For example, M1, M2 is two messages with the same keyword but from different queues, and the generation time is T1 and T2 respectively (T1< T2), then if the subscriber receives M1 first, then it will receive M2 later, and will transmit M2 as the final message to the upper layer application; if the subscriber first receives M2, M1 may be rejected and M2 may be passed to the upper layer application as the final message.
Example two
Referring to fig. 4, the basic steps of a method for implementing a subscription mode based on keyword filtering according to this embodiment are the same as those in the first embodiment, except that, as another implementation, when the number of message brokers is more than one, and one of the message brokers serves as a master node, and the remaining message brokers serve as slave nodes, the method further includes the following operations:
when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; after receiving the new message sent by the publisher, the subsequent master node forwards the new message to the slave node, and the slave node processes the new message according to the operation in the step B;
when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message according to the operation of step A, B, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message according to the operation of step C. It can be understood that, instead of the master node processing the slave nodes based on the keyword filtering, one of the plurality of slave nodes may be selected as the substitute master node, or the master node may be replaced by a plurality of slave nodes, and the specific application may be set according to an actual requirement, which is not specifically limited in this embodiment.
More specifically, in practical applications, the specific steps of the above operations may be as follows:
s1, when the slave node is started normally, inquiring the subsequent message increment from the master node according to the offset of the last message of each queue recorded by the slave node; synchronously updating the data in each queue and the container capable of being persisted according to the result returned by the main node;
s2, after receiving the message sent by the publisher, the subsequent host node processes the message according to the operation of the step B; the message is forwarded to the slave node, and the slave node processes the message according to the operation of the step B;
s3, when the master node fails and stops working (crashes) and the slave node for replacing the master node takes effect, the publisher sends a message to the slave node, and the slave node receives and processes the message according to the operation of step A, B; the subscriber sends the request of pulling the message to the slave node for processing, and the slave node pulls the message according to the operation of the step C.
It can be understood that after the master node recovers to normal operation, the master node performs data synchronization with the slave node used for replacing the master node, so that the data of the master node and the slave node can ensure final consistency, and the processing of subscribers is not affected.
EXAMPLE III
Based on the same inventive concept, referring to fig. 5, an embodiment of the present invention further provides a system for implementing a subscription mode based on keyword filtering, including a publisher, a message broker, and a subscriber. Wherein the message broker comprises: the system comprises a message receiving module, a message subject identification module, a message processing module and a message pulling module.
A message receiving module to: receiving a message which is issued by a publisher and specifies a message subject; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
a message topic identification module to: judging whether to filter keywords according to the identification prefix of the message theme;
a message processing module to: storing the received messages into the tail parts of one or more queues corresponding to the message subjects; when the keyword filtering is needed, the message is transferred to a durable container for filtering;
a message pull module to: when a request of pulling a message by a subscriber is received and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence-capable container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber.
Further, in practical applications, the message processing module stores the message in a persistent container for filtering, and specifically includes the following operations: when the keyword filtering is determined to be needed, transferring the information of the message to a persistence container, wherein the information of the message comprises: the method comprises the steps of generating a message queue, wherein the message queue comprises key words of the message, complete content of the message, an identification number of the queue where the message is located, offset in the message queue and a timestamp of the message; before storing, checking whether the information of the message corresponding to the keyword exists, and if so, covering the original message with a new message.
Further, in practical applications, the message pulling module pulls the filtered message back to the subscriber in the persistency container and/or the corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber, and specifically includes the following operations:
firstly, according to the message receiving base line of the subscriber, determining the position of the message to be pulled: (1) if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue; (2) if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled from the corresponding topic in the persistence container, and returning a mark pulled from the queue to the subscriber after judging that the data is completely taken from the persistence container according to the request condition of the subscriber; then pulling from the last message in the corresponding queue; (3) if the message receiving baseline of the subscriber is the specified baseline of the subscriber, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
and then according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled: (1) if the subscriber does not give the keyword, pulling all the messages at the pulling position and the subsequent positions; (2) if the subscriber gives the specified keyword, the message which contains the specified keyword and is at the pulling position and then is pulled.
Still further, in an optional implementation manner, the subscriber is provided with a message filtering module, configured to: and after receiving the message returned by the message agent, taking the message with the latest timestamp as a final message for the message with the same keyword and different queues, and transmitting the final message to the upper application.
Example four
The basic structure of the subscription mode implementation system based on keyword filtering provided by this embodiment is the same as that of the embodiment, and the difference is that: the system comprises more than one message agent, wherein one message agent is used as a main node, and the other message agents are used as slave nodes; when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; the subsequent main node receives the new message sent by the publisher and forwards the new message to the slave node, and the slave node processes the message by using the message subject identification module and the message processing module; when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message by using the message receiving module, the message subject identification module and the message processing module, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message by using the message pulling module.
Note that: the above-described embodiments are merely examples and are not intended to be limiting, and those skilled in the art can combine and combine some steps and devices from the above-described separately embodiments to achieve the effects of the present invention according to the concept of the present invention, and such combined and combined embodiments are also included in the present invention, and such combined and combined embodiments are not described herein separately.
Advantages, effects, and the like, which are mentioned in the embodiments of the present invention, are only examples and are not limiting, and they cannot be considered as necessarily possessed by the various embodiments of the present invention. Furthermore, the foregoing specific details disclosed herein are merely exemplary and for purposes of explanation and are not limiting, but rather the embodiments of the invention are set forth in order to provide a thorough understanding.
The block diagrams of devices, apparatuses, systems involved in the embodiments of the present invention are only given as illustrative examples, and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably herein. As used in connection with embodiments of the present invention, the terms "or" and "refer to the term" and/or "and are used interchangeably herein unless the context clearly dictates otherwise. The word "such as" is used with respect to embodiments of the present invention is intended to mean, and is used interchangeably with, the word "such as but not limited to".
The flow charts of steps in the embodiments of the present invention and the above description of the methods are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to an element in the singular, for example, using the articles "a," "an," or "the" is not to be construed as limiting the element to the singular.
In addition, the steps and devices in the embodiments of the present invention are not limited to be implemented in a certain embodiment, and in fact, some steps and devices in the embodiments of the present invention may be combined according to the concept of the present invention to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.
The respective operations in the embodiments of the present invention may be performed by any appropriate means capable of performing the corresponding functions. The means may include various hardware and/or software components and/or modules, including, but not limited to, hardware circuitry or a processor.
The method of an embodiment of the invention includes one or more acts for implementing the method described above. The methods and/or acts may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and/or use of specific actions may be modified without departing from the scope of the claims.
The functions in the embodiments of the present invention may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions on a tangible computer-readable medium. A storage media may be any available tangible media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk (disk) and Disc (Disc) include Compact Disc (CD), laser Disc, optical Disc, DVD (Digital Versatile Disc), floppy disk and blu-ray Disc where disks reproduce data magnetically, while discs reproduce data optically with lasers.
Accordingly, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions stored (and/or encoded) thereon that are executable by one or more processors to perform the operations described herein. The computer program product may include packaged material.
Other examples and implementations are within the scope and spirit of the embodiments of the invention and the following claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard-wired, or any combination of these. Features implementing functions may also be physically located at various locations, including being distributed such that portions of functions are implemented at different physical locations.
Various changes, substitutions and alterations to the techniques described herein may be made by those skilled in the art without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the claims of the present disclosure is not limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the invention to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof. And those not described in detail in this specification are within the skill of the art.
Claims (8)
1. A method for realizing subscription mode based on keyword filtering is characterized by comprising the following steps:
A. the message broker receives a message of a specified message topic issued by a publisher; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
B. the message agent stores the received message into the tail part of one or more queues corresponding to the message subject; judging whether to filter keywords according to the identification prefix of the message theme, if so, transferring the message to a durable container for filtering;
C. when the message agent receives a request of pulling a message from a subscriber and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistence container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber;
in the step B, whether to filter the keyword is determined according to the identification prefix of the message topic, and if yes, the message is transferred to a container capable of persisting for filtering, which specifically includes the following operations:
judging whether to filter keywords according to the identification prefix of the message theme, if so, continuously judging whether information of the message corresponding to the keywords is stored in the persistence container, if so, deleting the information and then inserting the information of the message into the persistence container, otherwise, directly inserting the information of the message into the persistence container; the information of the message includes: a key of the message, the complete content of the message, an identification number of the queue in which the message is located, an offset in the message re-queue, and a timestamp of the message.
2. The method for implementing a subscription mode based on keyword filtering according to claim 1, wherein in the step C, according to a message acceptance baseline condition of the subscriber and a keyword condition of a message given by the subscriber, in a persistence-capable container and/or a corresponding queue, the filtered message is pulled and returned to the subscriber, which specifically includes the following steps:
1) firstly, according to the message receiving base line of the subscriber, determining the position of the message to be pulled:
if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue;
if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled from the corresponding topic in the persistence container, and returning a mark pulled from the queue to the subscriber after judging that the data is completely taken from the persistence container according to the request condition of the subscriber; then pulling from the last message in the corresponding queue;
if the message receiving baseline of the subscriber is the specified baseline of the subscriber, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
2) and then according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled:
if the subscriber does not give the keyword, pulling all the messages at the pulling position and the subsequent positions;
if the subscriber gives the specified keyword, the message which contains the specified keyword and is at the pulling position and then is pulled.
3. The method for implementing a subscription mode based on keyword filtering as claimed in claim 1, wherein the method further comprises the following operations: after receiving the message returned by the message agent, the subscriber takes the message with the latest timestamp as the final message for the message with the same keyword and different queues, and transmits the final message to the upper application.
4. The method for implementing subscription mode based on keyword filtering as claimed in claim 1, wherein when the message brokers are set to more than one, and one of the message brokers is as the master node, and the rest of the message brokers are as the slave nodes, the method further comprises the following operations:
when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; after receiving the new message sent by the publisher, the subsequent master node forwards the new message to the slave node, and the slave node processes the new message according to the operation in the step B;
when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message according to the operation of step A, B, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message according to the operation of step C.
5. A subscription mode implementation system based on keyword filtering comprises a publisher, a message broker and a subscriber, and is characterized in that: the message agent comprises a message receiving module, a message subject identification module, a message processing module and a message pulling module;
the message receiving module is configured to: receiving a message which is issued by a publisher and specifies a message subject; the message body of the message carries the keywords of the message, and the message subject supporting the keyword filtering has an identification prefix;
the message topic identification module is configured to: judging whether to filter keywords according to the identification prefix of the message theme;
the message processing module is configured to: storing the received message into the tail part of one or more queues corresponding to the message subject; when the keyword filtering is needed, the message is transferred to a durable container for filtering;
the message pulling module is configured to: when a request of pulling a message by a subscriber is received and the message topic of the message supports keyword filtering, the filtered message is pulled and returned to the subscriber in a persistent container and/or a corresponding queue according to the message receiving baseline condition of the subscriber and the keyword condition of the message given by the subscriber;
the message processing module is used for transferring the message to a persistence container for filtering, and specifically comprises the following operations: when the keyword filtering is judged to be needed, transferring the information of the message to a persistence-capable container, wherein the information of the message comprises the following information: the method comprises the steps of generating a message queue, wherein the message queue comprises key words of the message, complete content of the message, an identification number of the queue where the message is located, offset in the message queue and a timestamp of the message; before storing, checking whether the information of the message corresponding to the keyword exists, and if so, covering the original message with a new message.
6. The system for implementing a subscription model based on keyword filtering as claimed in claim 5, wherein the message pull module pulls the filtered message back to the subscriber in the persisted container and/or the corresponding queue according to the baseline condition of the message accepted by the subscriber and the keyword condition of the message given by the subscriber, specifically comprising the following operations:
firstly, according to the message receiving base line of the subscriber, determining the position of the message to be pulled: if the message receiving baseline of the subscriber is from the current time, determining the position of the pull message as the position of the pull message from the last message in the corresponding queue; if the message receiving baseline of the subscriber is uncertain, determining the position of the pull message as that the message is pulled from the corresponding topic in the persistence container, and returning a mark pulled from the queue to the subscriber after judging that the data is completely taken from the persistence container according to the request condition of the subscriber; then pulling from the last message in the corresponding queue; if the message receiving baseline of the subscriber is the subscriber specified baseline, determining the position of the pull message as the pull message from the specified baseline in the corresponding queue;
and then according to the keywords of the messages to be pulled given by the subscriber, determining the range of the messages to be pulled: if the subscriber does not give the keyword, pulling all the messages at the pulling position and the subsequent positions; if the subscriber gives the specified keyword, the message containing the specified keyword and at the pulling position and after is pulled.
7. The system for implementing a subscription model based on keyword filtering as claimed in claim 5, wherein said subscriber is provided with a message filtering module for: and after receiving the message returned by the message agent, taking the message with the latest timestamp as a final message for the message with the same keyword and different queues, and transmitting the final message to the upper application.
8. The keyword filtering-based subscription pattern implementing system of claim 5, wherein: the system comprises more than one message agent, wherein one message agent is used as a main node, and the other message agents are used as slave nodes;
when the slave node is started, the data in the queue and the container which can be persisted are synchronized on the slave master node; the subsequent main node receives the new message sent by the publisher and forwards the new message to the slave node, and the slave node processes the message by using the message subject identification module and the message processing module;
when the master node fails and stops working, the publisher sends a message to the slave node, the slave node receives and processes the message by using the message receiving module, the message subject identification module and the message processing module, the subscriber sends a request for pulling the message to the slave node, and the slave node pulls the message by using the message pulling module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010285711.3A CN113542314B (en) | 2020-04-13 | 2020-04-13 | Method and system for realizing subscription mode based on keyword filtering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010285711.3A CN113542314B (en) | 2020-04-13 | 2020-04-13 | Method and system for realizing subscription mode based on keyword filtering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113542314A CN113542314A (en) | 2021-10-22 |
| CN113542314B true CN113542314B (en) | 2022-07-08 |
Family
ID=78119861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010285711.3A Active CN113542314B (en) | 2020-04-13 | 2020-04-13 | Method and system for realizing subscription mode based on keyword filtering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113542314B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114125057B (en) * | 2022-01-26 | 2022-04-26 | 中科边缘智慧信息科技（苏州）有限公司 | Distributed data interaction method and device for multi-service center |
| CN114666292A (en) * | 2022-04-01 | 2022-06-24 | 广州大学 | Instant messaging technology method, system and medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109428922A (en) * | 2017-08-29 | 2019-03-05 | 华为技术有限公司 | A kind of subscription dissemination method and server |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7293038B2 (en) * | 2003-02-25 | 2007-11-06 | Bea Systems, Inc. | Systems and methods for client-side filtering of subscribed messages |
| US7543068B2 (en) * | 2004-08-26 | 2009-06-02 | At&T Intellectual Property I, Lp | Filtering information at a data network based on filter rules associated with consumer processing devices |
| CN104092717B (en) * | 2013-12-13 | 2015-08-12 | 深圳市腾讯计算机系统有限公司 | Message treatment method and system, message destination equipment |
| CN104754036A (en) * | 2015-03-06 | 2015-07-01 | 合一信息技术（北京）有限公司 | Message processing system and processing method based on kafka |
| CN108076103A (en) * | 2016-11-18 | 2018-05-25 | 北京优朋普乐科技有限公司 | A kind of message treatment method, system and message producer's cluster |
| CN107656825A (en) * | 2017-09-01 | 2018-02-02 | 上海艾融软件股份有限公司 | Message treatment method, apparatus and system |
| US11431662B2 (en) * | 2017-12-28 | 2022-08-30 | Meta Platforms, Inc. | Techniques for message deduplication |
| CN108183967B (en) * | 2018-01-16 | 2020-08-04 | 重庆邮电大学 | OPC UA (optical proximity correction) publishing/subscribing method suitable for IPv6 wireless sensor network |
| CN108769099B (en) * | 2018-04-02 | 2021-09-07 | 武汉斗鱼网络科技有限公司 | Message de-duplication realization method of message middleware |
| US20190318036A1 (en) * | 2018-04-11 | 2019-10-17 | Wind River Systems, Inc. | Topic Based Publish and Parametric Subscribe Pattern |
| CN109194736B (en) * | 2018-08-30 | 2021-04-27 | 百度在线网络技术（北京）有限公司 | Message duplicate removal method and device, electronic equipment, medium and unmanned vehicle |
-
2020
- 2020-04-13 CN CN202010285711.3A patent/CN113542314B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109428922A (en) * | 2017-08-29 | 2019-03-05 | 华为技术有限公司 | A kind of subscription dissemination method and server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113542314A (en) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113542314B (en) | Method and system for realizing subscription mode based on keyword filtering | |
| CN111600965B (en) | Consensus method and system in blockchain | |
| CN110601903B (en) | Data processing method and device based on message queue middleware | |
| CN109376197B (en) | Data synchronization method, server and computer storage medium | |
| CN110045912A (en) | Data processing method and device | |
| CN102609479B (en) | A kind of memory database node clone method | |
| CN110392120B (en) | Method and device for recovering fault in message pushing process | |
| KR101545626B1 (en) | System for interoperation between dds and dbms | |
| CN112804332B (en) | Message processing system, method, device, equipment and computer readable storage medium | |
| CN110457141B (en) | Service message processing method, system, device and equipment | |
| WO2013163864A1 (en) | Data persistence processing method and device and database system | |
| CN108228581B (en) | Zookeeper compatible communication method, server and system | |
| WO2022001750A1 (en) | Data synchronization processing method, electronic device, and storage medium | |
| CN102438042B (en) | Dynamic parameter synchronizing method and system of multipoint access device | |
| CN106802892A (en) | For the method and apparatus of master/slave data consistency verification | |
| CN110532123A (en) | The failover method and device of HBase system | |
| US20140108484A1 (en) | Method and system for optimizing distributed transactions | |
| CN111416823A (en) | Data transmission method and device | |
| CN115146002A (en) | Cross-data-center data synchronization method and device | |
| CN112822091A (en) | Message processing method and device | |
| CN111143126A (en) | Data copying method, system and related components of distributed file system | |
| CN114253929A (en) | Network disk system architecture based on distributed file storage | |
| CN110417892A (en) | Data Replication Link optimization method and device based on packet parsing | |
| CN107846476A (en) | A kind of information synchronization method, equipment and storage medium, program product | |
| CN114051036A (en) | Data synchronization method, device and equipment for rail transit signal system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |