CN111611090A - Distributed message processing method and system - Google Patents
Distributed message processing method and system Download PDFInfo
- Publication number
- CN111611090A CN111611090A CN202010401681.8A CN202010401681A CN111611090A CN 111611090 A CN111611090 A CN 111611090A CN 202010401681 A CN202010401681 A CN 202010401681A CN 111611090 A CN111611090 A CN 111611090A
- Authority
- CN
- China
- Prior art keywords
- message
- receiving
- timestamp
- messages
- overtime
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a distributed message processing method and a system, wherein the method comprises the following steps: receiving a message sending request sent by a client, generating a message containing a sending time stamp according to the message sending request, adding the message into a corresponding message queue, wherein the sending time stamp has uniqueness in the message queue, and arranging the messages in the message queue based on the sending time stamp; judging whether the message is overtime, marking the overtime message according to the judgment result and backing up the overtime message to a distributed file system; and periodically deleting the overtime messages in the message queue according to the preset clearing time. The invention ensures the order of message transmission by designing the sending time stamp; by designing the steps of judging and clearing the overtime message, the distributed file system and the message queue can be matched with each other, so that the client can inquire the message and simultaneously reduce the data storage pressure brought by the message queue.
Description
Technical Field
The invention relates to the technical field of computer software, in particular to a distributed message processing method and system.
Background
The Hadoop platform is the most famous big data platform, a big data processing technology stack is constructed based on Hadoop at present, and most application programs based on the big data platform need to process distributed messages;
however, the existing message processing system is too specialized/complicated or too simple, such as microsoft MSMQ is an industrial level message processing system with powerful functions and hard delivery guarantee, but deployment of the system requires a proprietary system of microsoft, which is high in cost; amazon, for example, provides a Simple Queue Service (SQS) with low cost, which has only basic message passing operations (sending/receiving), and cannot guarantee the order of message passing.
Therefore, when the Hadoop user is provided with the message queue function for guaranteeing the message passing order nowadays, an extra message processing system which is too specialized/complicated nowadays is often required to be deployed, which is too high in cost and will increase the complexity of the corresponding application program.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a distributed message processing method and a distributed message processing system which can transmit messages in sequence and have low deployment cost.
In order to solve the technical problem, the invention is solved by the following technical scheme:
a distributed message processing method, comprising the steps of:
receiving a message sending request sent by a client, generating a message containing a sending time stamp according to the message sending request, adding the message into a corresponding message queue, wherein the sending time stamp has uniqueness in the message queue, and arranging the messages in the message queue based on the sending time stamp;
judging whether the message is overtime, marking the overtime message according to the judgment result and backing up the overtime message to a distributed file system;
and periodically deleting the overtime messages in the message queue according to the preset clearing time.
As an implementable embodiment, after adding the message to the corresponding message queue, the method further includes a message transmission step, and the specific steps are as follows:
receiving a message receiving request sent by a client, scanning messages in a message queue and/or a distributed file system based on the message receiving request, and obtaining a first message to be transmitted;
and adding delivery time stamps to the first message to be delivered in sequence based on the sending time stamps, marking the first message to be delivered as delivered and sending the first message to the client.
As an implementable embodiment:
receiving a message receiving request sent by a client, wherein the message receiving request comprises a latest receiving timestamp;
acquiring a current receiving timestamp based on the message receiving request and feeding back the current receiving timestamp, so that the client updates the latest receiving timestamp according to the current receiving timestamp;
calculating time to be scanned based on the latest receiving timestamp, judging whether overtime messages need to be scanned based on the time to be scanned, scanning messages of a message queue based on the latest receiving timestamp when the overtime messages do not need to be scanned, and otherwise, scanning messages of the message queue and messages in a distributed file system based on the latest receiving timestamp to obtain a second message to be transmitted;
the first message to be transferred is extracted from the second message to be transferred.
As an implementable embodiment:
when the last receive timestamp is empty, all messages in the message queue and the distributed file system are scanned.
As an implementable embodiment:
the message sending request includes a delivery guarantee;
obtaining the transmission guarantee of a second message to be transmitted, detecting the transmission condition of the second message to be transmitted, and obtaining a detection result;
and extracting the first message to be transferred from the second message to be transferred based on the transfer guarantee and the detection result.
As an implementable embodiment:
the transmission timestamp includes a time, a random number, and an ID of a corresponding timestamp management node.
As an implementable embodiment:
the messages in the message queue are stored in the embedded database in a distributed mode.
The invention also provides a distributed message processing system which is connected with the distributed file system and comprises a message management node, a timestamp management node and a client;
the message management node includes:
the message publishing module is used for receiving a message sending request sent by a client, generating a message containing a sending time stamp according to the message sending request, and adding the message into a corresponding message queue, wherein the sending time stamp is sent by a corresponding time stamp management node, and the sending time stamp has uniqueness in the message queue; the message publishing module is further configured to arrange the messages in the message queue based on the sending timestamps;
the overtime judging module is used for judging whether the message is overtime or not, marking the overtime message according to the judging result and backing up the overtime message to the distributed file system;
and the overtime clearing module is used for periodically deleting the overtime messages in the message queue according to the preset clearing time.
As an implementable embodiment:
the message management node further comprises a message passing module;
the message transmission module is used for receiving a message receiving request sent by a client, and scanning messages in a message queue and/or a distributed file system based on the message receiving request to obtain a first message to be transmitted; and the system is also used for adding delivery time stamps to the first message to be delivered in sequence based on the sending time stamps, marking the first message to be delivered as delivered and sending the first message to the client.
As an implementable embodiment:
the message transmission module comprises a receiving unit, a timestamp updating unit, a scanning unit, an extracting unit and a transmitting unit;
the receiving unit is used for receiving a message receiving request sent by a client, wherein the message receiving request comprises a latest receiving timestamp;
the timestamp updating unit is used for acquiring and feeding back a current receiving timestamp from a corresponding timestamp management node based on the message receiving request, so that the client updates the latest receiving timestamp according to the current receiving timestamp;
the scanning unit is used for calculating time to be scanned based on the latest receiving timestamp, judging whether overtime messages need to be scanned based on the time to be scanned, scanning messages of the message queue based on the latest receiving timestamp when the overtime messages do not need to be scanned, and otherwise, scanning messages of the message queue and the distributed file system based on the latest receiving timestamp to obtain second messages to be transmitted;
the extracting unit is used for extracting the first message to be transmitted from the second message to be transmitted.
Due to the adoption of the technical scheme, the invention has the remarkable technical effects that:
according to the message sending method and the message sending system, the message containing the sending time stamp is generated according to the message sending request, and the sending time stamp has uniqueness in the message queue, so that the messages in the message queue can be arranged based on the sending time stamp, and a client receives the messages according to the total sequence specified by the sending time stamp when receiving the messages, thereby ensuring the message transmission sequence; meanwhile, through the steps of judging and clarifying the overtime information, the distributed file system and the message queue can be matched with each other, so that the client can conveniently inquire the message and simultaneously reduce the data storage pressure brought by the message queue; compared with the prior art, the method provided by the invention is used for processing the messages, and only the embedded database needs to be arranged in the cluster of the existing distributed file system on the premise of ensuring the messages to be transmitted in sequence, so that the huge cost is not needed, and the attribute of the distributed file system is not changed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow diagram of a distributed message processing method of the present invention;
FIG. 2 is a block diagram of a distributed message processing system according to the present invention;
fig. 3 is a schematic block diagram of the message management node 100 in fig. 2.
Detailed Description
The present invention will be described in further detail with reference to examples, which are illustrative of the present invention and are not to be construed as being limited thereto.
Embodiment 1, a distributed message processing method, as shown in fig. 1, includes the following steps:
s100, receiving a message sending request sent by a client 200, generating a message containing a sending time stamp according to the message sending request, adding the message into a corresponding message queue, wherein the sending time stamp has uniqueness in the message queue, and arranging the messages in the message queue based on the sending time stamp;
in this embodiment, a message line is written in a corresponding message queue in advance based on a message sending request, a sending timestamp is requested from the corresponding timestamp management node 300, the message is generated after the obtained sending timestamp is inserted into the message line, and the message is marked as sent at this time, so that the message is sent based on the message sending request.
In the actual use process, the client 200 creates or joins the message queue in advance, so that the message can be sent to the message queue or received.
In this embodiment, the message sending request includes the client 200ID, the queue ID, the message body, and the message type, and may further include the client 200ID of the recipient, where both the client 200ID and the queue ID have uniqueness; the message body is message content or a pointer pointing to a file, and only a sender/receiver knows how to acquire the file; the message type is data type information corresponding to the message body. The sender/recipient realizes messaging through their client 200.
Note: creating a message queue based on the request of the client 200, adding the client 200 to an existing message queue, sending a message, receiving a message, and the like are conventional technologies in the art, so detailed description of specific implementation steps thereof is not given in this embodiment.
As can be seen from the above, in this embodiment, a message including a sending timestamp is generated according to the message sending request, and the sending timestamp has uniqueness in the message queue, so that messages in the message queue can be arranged based on the sending timestamp, and when receiving a message, the client 200 receives messages according to a total sequence specified by the sending timestamp, thereby ensuring a message transmission sequence.
S200, judging whether the message is overtime, marking the overtime message according to the judgment result and backing up the overtime message to the distributed file system 400;
in this embodiment, the distributed file system 400 employs a hadoop file system (HDFS);
the specific method for judging whether the message is overtime comprises the following steps:
acquiring the waiting time of the message based on the sending timestamp of the message, judging that the message is overtime when the waiting time reaches a preset overtime threshold (2min), marking the message as an overtime message and backing up the overtime message to the distributed file system 400;
the above message queues are represented by a Hadoop HDFS system table named with queues, i.e., each message queue is an HDFS file. In the embodiment, the messages in the message queue are stored in the embedded database in a distributed manner.
And S300, periodically deleting the overtime messages in the message queue according to the preset clearing time.
The clearing time can be set by a person skilled in the art, and in this embodiment, the clearing time is 2min, that is, the timeout message in the message queue is cleared every 2 min.
Therefore, the message queue is actually a temporary table, and is used for implementing real-time reading and writing of a recent active message (a message within a timeout threshold), and when the message is overtime, the message is backed up to the distributed file system 400, so that subsequent query of the message is facilitated.
Since the distributed file system 400 often stores a large amount of messages, for example, the scanning time is too long when the distributed file system 400 scans messages to be received by the client 200 directly, and all the messages are retained in the message queue, which causes too much pressure for storing the message queue, the judgment and removal of the overtime message in the embodiment can make the distributed file system 400 and the message queue cooperate with each other, so that the data storage pressure caused by the message queue is reduced while the client 200 queries the messages.
Compared with the prior art, the method provided by the embodiment is used for processing the messages, and only the embedded database needs to be deployed in the cluster of the existing distributed file system 400 on the premise of ensuring the messages to be transmitted in sequence, so that huge cost is not needed, and the properties of the Hadoop file system, such as scalability, fault tolerance and the like, are not changed.
Further, after the step S100 adds the message to the corresponding message queue, the method further includes a message transmitting step, and the specific steps are as follows:
s400, receiving a message receiving request sent by the client 200, and scanning messages in a message queue and/or the distributed file system 400 based on the message receiving request to obtain a first message to be transmitted;
the concrete implementation steps are as follows:
s410, receiving a message receiving request sent by the client 200, wherein the message receiving request comprises a latest receiving time stamp;
the message receiving request also includes information such as client 200ID, queue ID, etc., which belongs to the prior conventional technology, so the specification does not additionally inform.
The latest reception timestamp is used to indicate the time when the client 200 last initiated a message reception request.
S420, acquiring and feeding back a current receiving time stamp based on the message receiving request, so that the client 200 updates the latest receiving time stamp according to the current receiving time stamp;
that is, the current receiving timestamp is used as the latest receiving timestamp reported when the client 200 initiates the message receiving request next time.
S430, calculating time to be scanned based on the latest receiving timestamp, judging whether overtime information needs to be scanned based on the time to be scanned, scanning information in an information queue based on the latest receiving timestamp when the overtime information does not need to be scanned, and otherwise, scanning information in the information queue and the distributed file system 400 based on the latest receiving timestamp to obtain a second message to be transmitted;
in this embodiment, by designing the latest receiving timestamp, only the messages in the time period from the last time the client 200 receives the message to the present are scanned, and it is not necessary to scan all the messages in the message queue and/or the distributed file system 400, so as to accelerate the scanning speed.
Since the message queue in this embodiment is equivalent to a temporary table, only messages that are active in the near term (i.e., messages that are not clear temporarily) are stored, and the timeout messages are saved as a copy of the HDFS with persistence in the distributed file system 400, which is convenient for the user to receive or query through the client 200; in this embodiment, the time length (time to be scanned) of the last message received by the client 200 is calculated according to the latest receiving timestamp, so as to determine whether the client needs to scan the timeout message, and the message queue and/or the message of the corresponding time interval in the distributed file system 400 are/is scanned based on the determination result.
The concrete implementation steps are as follows:
when the time to be scanned is less than the preset timeout threshold, that is, it indicates that none of the information to be scanned is timeout, it is determined that the message timeout needs not to be scanned, and at this time, the messages in the message queue (that is, the messages from the time point when the client 200 receives the message last time) are scanned based on the latest receiving timestamp.
When the time to be scanned is greater than or equal to the preset timeout threshold, which indicates that the scanning of the timeout message is required, the messages in the distributed file system 400 and the message queue are scanned based on the latest receiving timestamp.
During actual processing, the last received timestamp may be null, and when the last received timestamp is null, all messages in the message queue and the distributed file system 400 are scanned.
And taking the scanned message corresponding to the message receiving request as a second message to be transmitted.
And S440, extracting the first message to be transferred from the second message to be transferred.
Since there is a message that has timed out but is not cleared, so that there is a repeat message in the second to-be-transferred message, the embodiment deduplicates the second to-be-transferred message obtained by scanning to obtain the first to-be-transferred message.
Further, the message receipt request in this embodiment includes a delivery guarantee specifying the number of times the message should be delivered, in this embodiment specifying "at least once" or "at most once" the message should be delivered.
Step S440 of extracting the first message to be transferred from the second message to be transferred further includes a step of checking a transfer guarantee, which includes the following specific steps:
obtaining the transmission guarantee of a second message to be transmitted, detecting the transmission condition of the second message to be transmitted, and obtaining a detection result;
and extracting the first message to be transferred from the second message to be transferred based on the transfer guarantee and the detection result.
That is, when the transfer guarantee is "at most once", only the second to-be-transferred message that is not marked as transferred is transferred as the first to-be-transferred message.
S500, adding delivery time stamps to the first message to be delivered in sequence based on the sending time stamps, and marking the first message to be delivered as delivered and sending the first message to the client 200, that is, in the same message queue, delivering the message to the client 200 in the sequence of sending time stamps.
In this embodiment, the first message to be transmitted is marked as transmitted and then sent to the client 200, which can effectively keep the message guarantee of "at most once", otherwise, if the message is transmitted to the client 200 first and then the message is marked as transmitted according to the transmission result fed back by the client 200, there is a fault after the client 200 acquires the message, the message is marked as transmitted only after the client 200 is reactivated, and the same message may be transmitted again in the waiting process, thereby destroying the transmission guarantee of "at most once".
Further, the transmission timestamp includes time (in ms), a random number, and an ID of the corresponding timestamp management node 300.
In this embodiment, the format of the delivery timestamp and the current reception timestamp are the same as the transmission timestamp. Each timestamp management node 300 synchronizes time with an external time server based on the NTP protocol and ensures that the accuracy of the synchronization time is within the 200ms error range.
In this embodiment, the sending timestamps are required to be unique in the message queue, so that the messages in the message queue can be arranged and processed according to the order of the sending timestamps, and in actual use, different clients 200 can issue the messages at the same time, that is, the time for the clients 200 to write the messages is the same, for example, the timestamp indicating the current time is directly added to the message, so that the sending timestamps cannot meet the unique and strict ordering requirement.
Therefore, the format of the transmission timestamp is designed in the embodiment, the format is NTP time + ID of the corresponding timestamp management node 300 + random number, and the lengths of the NTP time, the ID of the corresponding timestamp management node 300 and the random number are fixed, so that even if a plurality of clients 200 issue messages at the same time and the timestamp management nodes 300 which distribute timestamps for the messages are the same, the uniqueness of the transmission timestamp can be ensured, the messages can be strictly sequenced based on the transmission timestamp, and the transmission timestamp is a long number, thereby facilitating the optimized expression.
Embodiment 2, a distributed message processing system, as shown in fig. 2, is connected to a distributed file system 400, and includes a plurality of message management nodes 100, a plurality of timestamp management nodes 300, and a plurality of clients 200;
as shown in fig. 3, the message management node 100 includes:
a message publishing module 110, configured to receive a message sending request sent by a client 200, generate a message including a sending timestamp according to the message sending request, add the message to a corresponding message queue, where the sending timestamp is issued by a corresponding timestamp management node 300, and the sending timestamp has uniqueness in the message queue; the message publishing module 110 is further configured to arrange the messages in the message queue based on the sending timestamp;
the timeout judging module 120 is configured to judge whether the message is timeout, mark a timeout message according to a judgment result, and backup the timeout message to the distributed file system 400;
and the timeout removing module 130 is configured to periodically delete the timeout messages in the message queue according to a preset removing time.
Further, the message management node 100 further comprises a message passing module;
the message passing module 140 is configured to receive a message receiving request sent by the client 200, and scan a message queue and/or messages in the distributed file system 400 based on the message receiving request to obtain a first message to be passed; and is further configured to add delivery timestamps to the first message to be delivered in sequence based on the sending timestamp, and mark the first message to be delivered as delivered and send it to the client 200.
Further, the message delivery module 140 includes a receiving unit, a timestamp updating unit, a scanning unit, an extracting unit, and a delivery unit;
the receiving unit is configured to receive a message receiving request sent by the client 200, where the message receiving request includes a latest receiving timestamp;
the timestamp updating unit is configured to obtain and feed back a current receiving timestamp from the corresponding timestamp management node 300 based on the message receiving request, so that the client 200 updates the latest receiving timestamp according to the current receiving timestamp;
the scanning unit is used for calculating the time to be scanned based on the latest receiving timestamp, judging whether the overtime message needs to be scanned based on the time to be scanned, scanning the message of the message queue based on the latest receiving timestamp when the overtime message does not need to be scanned, and otherwise, scanning the message queue and the messages in the distributed file system 400 based on the latest receiving timestamp to obtain a second message to be transmitted;
the extracting unit is used for extracting the first message to be transmitted from the second message to be transmitted.
The delivery unit is configured to add delivery timestamps to the first message to be delivered in sequence based on the sending timestamps, and mark the first message to be delivered as delivered and send the first message to the client 200.
Further, the extraction unit is configured to:
obtaining the transmission guarantee of a second message to be transmitted, detecting the transmission condition of the second message to be transmitted, and obtaining a detection result;
and extracting the first message to be transferred from the second message to be transferred based on the transfer guarantee and the detection result.
In this embodiment, the distributed file system 400 employs a hadoop file system (HDFS);
the message managing node 100 is also arranged to store message queues, i.e. each message managing node 100 constitutes a distributed message processor comprising an embedded database (the embedded schema of the H2 relational database).
Each timestamp management node 300 constitutes a distributed timestamp manager.
The message management node 100 and the timestamp management node 300 are both deployed in a machine of a cluster in which the distributed file system 400 is located.
In this embodiment, each message queue is stored by using a multi-layer data indexing architecture and a message data fragmentation method based on an embedded database, that is, the clustered machines include a first machine and a second machine, where the first machine is an elected meta machine and is responsible for indexing global data and informing a second machine in which data of each query corresponding message exists, and the second machine is used for storing messages.
In this embodiment, the existing hash algorithm is used to segment the message queues, that is, each message queue is determined to be segmented into hash blocks of a corresponding number according to the number of second machines in the cluster for storage.
In addition, in the design of clearing the timeout message in this embodiment, after the timeout message is cleared once, the stored message is automatically balanced based on the distribution of the first machine and the second machine in the cluster, so when the number of machines in the cluster changes, real-time balancing is not required, only a new hash block needs to be added, an original hash block does not need to be allocated, and data balancing is uniformly performed after the timeout message is cleared.
The timestamp management node 300 may adopt a timestamp management Agent (Agent), and a person skilled in the relevant art may statically/dynamically deploy the timestamp management node 300 according to actual conditions, so as to ensure that each large area (or rack) in the cluster has the same number of timestamp management nodes 300, and ensure that each sub-area in each large area (or rack) has the same number of timestamp management nodes 300. The static deployment is to deploy timestamp management agents (agents) in manually-assigned machines as timestamp management nodes 300, and the dynamic deployment is to automatically start the timestamp management agents (agents) in the machines with the lowest load according to the specified number and close the timestamp management agents (agents) in other machines according to the load conditions of the machines in each sub-domain.
The distributed message processing system proposed in this embodiment has the following four failures in application, namely server computer failure, network failure, client 200 failure, and suspicious failure. The System can process the computer fault and the network fault of the front server end by means of a Hadoop Distributed File System 400 (HDFS). For client 200 failures and suspected failures, one skilled in the art can set the handling mechanism of message sending failure by itself, such as using an atomic method to ensure that a message is sent or never sent.
In summary, the present embodiment can sequentially transfer the messages in the message queue only by deploying the distributed message processor and the distributed timestamp manager, which has a low deployment cost compared to the existing message processing system such as microsoft MSMQ, and when microsoft MSMQ is used to process the messages, the corresponding application program must be able to process the fault caused by both/any one of MSMQ and/or Hadoop at the same time.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should be noted that:
reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrase "one embodiment" or "an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
In addition, it should be noted that the specific embodiments described in the present specification may differ in the shape of the components, the names of the components, and the like. All equivalent or simple changes of the structure, the characteristics and the principle of the invention which are described in the patent conception of the invention are included in the protection scope of the patent of the invention. Various modifications, additions and substitutions for the specific embodiments described may be made by those skilled in the art without departing from the scope of the invention as defined in the accompanying claims.
Claims (10)
1. A distributed message processing method, comprising the steps of:
receiving a message sending request sent by a client, generating a message containing a sending time stamp according to the message sending request, adding the message into a corresponding message queue, wherein the sending time stamp has uniqueness in the message queue, and arranging the messages in the message queue based on the sending time stamp;
judging whether the message is overtime, marking the overtime message according to the judgment result and backing up the overtime message to a distributed file system;
and periodically deleting the overtime messages in the message queue according to the preset clearing time.
2. The distributed message processing method according to claim 1, wherein after adding the message to the corresponding message queue, further comprising a message transmitting step, specifically comprising:
receiving a message receiving request sent by a client, scanning messages in a message queue and/or a distributed file system based on the message receiving request, and obtaining a first message to be transmitted;
and adding delivery time stamps to the first message to be delivered in sequence based on the sending time stamps, marking the first message to be delivered as delivered and sending the first message to the client.
3. The distributed message processing method of claim 2, wherein:
receiving a message receiving request sent by a client, wherein the message receiving request comprises a latest receiving timestamp;
acquiring a current receiving timestamp based on the message receiving request and feeding back the current receiving timestamp, so that the client updates the latest receiving timestamp according to the current receiving timestamp;
calculating time to be scanned based on the latest receiving timestamp, judging whether overtime messages need to be scanned based on the time to be scanned, scanning messages of a message queue based on the latest receiving timestamp when the overtime messages do not need to be scanned, and otherwise, scanning messages of the message queue and messages in a distributed file system based on the latest receiving timestamp to obtain a second message to be transmitted;
the first message to be transferred is extracted from the second message to be transferred.
4. The distributed message processing method of claim 3, wherein:
when the last receive timestamp is empty, all messages in the message queue and the distributed file system are scanned.
5. The distributed message processing method of claim 4, wherein:
the message sending request includes a delivery guarantee;
obtaining the transmission guarantee of a second message to be transmitted, detecting the transmission condition of the second message to be transmitted, and obtaining a detection result;
and extracting the first message to be transferred from the second message to be transferred based on the transfer guarantee and the detection result.
6. The distributed message processing method of any of claims 1 to 5, characterized by:
the transmission timestamp includes a time, a random number, and an ID of a corresponding timestamp management node.
7. The distributed message processing method of any of claims 1 to 5, characterized by:
the messages in the message queue are stored in the embedded database in a distributed mode.
8. A distributed message processing system is connected with a distributed file system and is characterized by comprising a message management node, a timestamp management node and a client;
the message management node includes:
the message publishing module is used for receiving a message sending request sent by a client, generating a message containing a sending time stamp according to the message sending request, and adding the message into a corresponding message queue, wherein the sending time stamp is sent by a corresponding time stamp management node, and the sending time stamp has uniqueness in the message queue; the message publishing module is further configured to arrange the messages in the message queue based on the sending timestamps;
the overtime judging module is used for judging whether the message is overtime or not, marking the overtime message according to the judging result and backing up the overtime message to the distributed file system;
and the overtime clearing module is used for periodically deleting the overtime messages in the message queue according to the preset clearing time.
9. A distributed message processing system as defined in claim 8, wherein:
the message management node further comprises a message passing module;
the message transmission module is used for receiving a message receiving request sent by a client, and scanning messages in a message queue and/or a distributed file system based on the message receiving request to obtain a first message to be transmitted; and the system is also used for adding delivery time stamps to the first message to be delivered in sequence based on the sending time stamps, marking the first message to be delivered as delivered and sending the first message to the client.
10. A distributed message processing system as defined in claim 9, wherein:
the message transmission module comprises a receiving unit, a timestamp updating unit, a scanning unit, an extracting unit and a transmitting unit;
the receiving unit is used for receiving a message receiving request sent by a client, wherein the message receiving request comprises a latest receiving timestamp;
the timestamp updating unit is used for acquiring and feeding back a current receiving timestamp from a corresponding timestamp management node based on the message receiving request, so that the client updates the latest receiving timestamp according to the current receiving timestamp;
the scanning unit is used for calculating time to be scanned based on the latest receiving timestamp, judging whether overtime messages need to be scanned based on the time to be scanned, scanning messages of the message queue based on the latest receiving timestamp when the overtime messages do not need to be scanned, and otherwise, scanning messages of the message queue and the distributed file system based on the latest receiving timestamp to obtain second messages to be transmitted;
the extracting unit is used for extracting the first message to be transmitted from the second message to be transmitted.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111463171.4A CN114138520B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
| CN202010401681.8A CN111611090B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010401681.8A CN111611090B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111463171.4A Division CN114138520B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111611090A true CN111611090A (en) | 2020-09-01 |
| CN111611090B CN111611090B (en) | 2021-12-28 |
Family
ID=72196925
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010401681.8A Active CN111611090B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
| CN202111463171.4A Active CN114138520B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111463171.4A Active CN114138520B (en) | 2020-05-13 | 2020-05-13 | Distributed message processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN111611090B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113778701A (en) * | 2021-01-07 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Message processing method and device, electronic device and medium |
| CN115878639A (en) * | 2022-09-07 | 2023-03-31 | 贝壳找房(北京)科技有限公司 | Consistency processing method of secondary cache and distributed service system |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104301203A (en) * | 2014-09-10 | 2015-01-21 | 腾讯科技（深圳）有限公司 | Message pushing method and equipment |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN107547605A (en) * | 2016-06-29 | 2018-01-05 | 华为技术有限公司 | A kind of message reading/writing method and node device based on node queue |
| US9894143B1 (en) * | 2013-11-06 | 2018-02-13 | Amazon Technologies, Inc. | Pre-processing and processing pipeline for queue client |
| US20180052722A1 (en) * | 2013-02-27 | 2018-02-22 | International Business Machines Corporation | Link optimization for callout request messages |
| CN108737208A (en) * | 2018-03-21 | 2018-11-02 | 北京天融信网络安全技术有限公司 | Connection synchronous method, device and computer based on security gateway deep-packet detection |
| CN109460438A (en) * | 2018-09-26 | 2019-03-12 | 中国平安人寿保险股份有限公司 | Message data storage method, device, computer equipment and storage medium |
| CN109558425A (en) * | 2018-11-19 | 2019-04-02 | 郑州云海信息技术有限公司 | A kind of backup method and device of caching |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| US10362131B1 (en) * | 2008-06-18 | 2019-07-23 | Amazon Technologies, Inc. | Fault tolerant message delivery |
| CN110392120A (en) * | 2019-08-15 | 2019-10-29 | 锐捷网络股份有限公司 | The restoration methods and device of failure during a kind of push of message |
| CN111104257A (en) * | 2019-11-30 | 2020-05-05 | 浪潮(北京)电子信息产业有限公司 | Anti-timeout method, device, equipment and medium for backup log data |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007024120A1 (en) * | 2005-08-26 | 2007-03-01 | Electronics And Telecommunications Research Institute | Method for requesting resource and scheduling for uplink traffic in mobile communication and apparatus thereof |
| CN101459627B (en) * | 2008-04-07 | 2012-09-05 | 中兴通讯股份有限公司 | Message management method |
| JP6253984B2 (en) * | 2010-09-10 | 2017-12-27 | ビジブル・テクノロジーズ・インコーポレイテッド | System and method for reputation management of consumer sent media |
| CN103019866B (en) * | 2012-10-24 | 2017-02-08 | 北京京东世纪贸易有限公司 | Distributed method and system based on message queue |
| US9614939B2 (en) * | 2014-05-08 | 2017-04-04 | Google Inc. | Network timeouts using intentionally delayed transmissions |
| US10230670B1 (en) * | 2014-11-10 | 2019-03-12 | Google Llc | Watermark-based message queue |
| US10581751B1 (en) * | 2015-12-16 | 2020-03-03 | EMC IP Holding Company LLC | Multi-queue based system for throttling backup agent save processes |
| CN106850397A (en) * | 2016-12-13 | 2017-06-13 | 深圳市智物联网络有限公司 | Message delivery method and device in Internet of Things |
| CN106789431B (en) * | 2016-12-26 | 2019-12-06 | 中国银联股份有限公司 | Overtime monitoring method and device |
| CN107181674A (en) * | 2017-06-16 | 2017-09-19 | 深圳市盛路物联通讯技术有限公司 | Message delivery method and device in Internet of Things |
| CN108009022A (en) * | 2017-11-06 | 2018-05-08 | 联动优势科技有限公司 | A kind of message treatment method and server |
| CN108959660B (en) * | 2018-08-15 | 2021-07-27 | 东北大学 | HDFS (Hadoop distributed file system) -based storage method and use method |
| US10805094B2 (en) * | 2018-10-08 | 2020-10-13 | International Business Machines Corporation | Blockchain timestamp agreement |
| CN109495375B (en) * | 2018-11-02 | 2021-04-13 | 广州小鹏汽车科技有限公司 | MQTT message processing method and device, electronic equipment and storage medium |
-
2020
- 2020-05-13 CN CN202010401681.8A patent/CN111611090B/en active Active
- 2020-05-13 CN CN202111463171.4A patent/CN114138520B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10362131B1 (en) * | 2008-06-18 | 2019-07-23 | Amazon Technologies, Inc. | Fault tolerant message delivery |
| US20180052722A1 (en) * | 2013-02-27 | 2018-02-22 | International Business Machines Corporation | Link optimization for callout request messages |
| US9894143B1 (en) * | 2013-11-06 | 2018-02-13 | Amazon Technologies, Inc. | Pre-processing and processing pipeline for queue client |
| CN104301203A (en) * | 2014-09-10 | 2015-01-21 | 腾讯科技（深圳）有限公司 | Message pushing method and equipment |
| CN107547605A (en) * | 2016-06-29 | 2018-01-05 | 华为技术有限公司 | A kind of message reading/writing method and node device based on node queue |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN108737208A (en) * | 2018-03-21 | 2018-11-02 | 北京天融信网络安全技术有限公司 | Connection synchronous method, device and computer based on security gateway deep-packet detection |
| CN109460438A (en) * | 2018-09-26 | 2019-03-12 | 中国平安人寿保险股份有限公司 | Message data storage method, device, computer equipment and storage medium |
| CN109558425A (en) * | 2018-11-19 | 2019-04-02 | 郑州云海信息技术有限公司 | A kind of backup method and device of caching |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN110392120A (en) * | 2019-08-15 | 2019-10-29 | 锐捷网络股份有限公司 | The restoration methods and device of failure during a kind of push of message |
| CN111104257A (en) * | 2019-11-30 | 2020-05-05 | 浪潮(北京)电子信息产业有限公司 | Anti-timeout method, device, equipment and medium for backup log data |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113778701A (en) * | 2021-01-07 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Message processing method and device, electronic device and medium |
| CN115878639A (en) * | 2022-09-07 | 2023-03-31 | 贝壳找房(北京)科技有限公司 | Consistency processing method of secondary cache and distributed service system |
| CN115878639B (en) * | 2022-09-07 | 2023-10-24 | 贝壳找房(北京)科技有限公司 | Consistency processing method of secondary cache and distributed service system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114138520B (en) | 2022-06-28 |
| CN111611090B (en) | 2021-12-28 |
| CN114138520A (en) | 2022-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107465767B (en) | Data synchronization method and system | |
| CN111611090B (en) | Distributed message processing method and system | |
| CN104092719B (en) | Document transmission method, device and distributed cluster file system | |
| US20040078440A1 (en) | High availability event topic | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| KR20080068110A (en) | A method for processing data synchronization and client terminal, server and data synchronization system thereof | |
| US20150256504A1 (en) | Distributed synchronization data in a message management service | |
| CN110324406B (en) | Method for acquiring business data and cloud service system | |
| CN105162879A (en) | Method, device and system for realizing data consistency among plurality of machine rooms | |
| US7788330B2 (en) | System and method for processing data associated with a transmission in a data communication system | |
| JP5065259B2 (en) | Apparatus, system, method, and computer program useful for communication between enterprise information system and client (device, system, and method useful for communication between enterprise information system and client) | |
| CN110071968B (en) | Block chain based message storage method and device | |
| EP1131756B1 (en) | Protocol for synchronizing parallel processors in a mobile communications system | |
| CN110958150A (en) | Management method and device for dynamic service configuration | |
| EP3687133B1 (en) | System and method for synchronization of media objects between devices operating in a multiroom system | |
| CN116405547A (en) | Message pushing method and device, processor, electronic equipment and storage medium | |
| CN111552719A (en) | Data management method, device and system, big data platform and readable storage medium | |
| CN110909048A (en) | Data query method, device, server, client and storage medium | |
| WO2007055867A1 (en) | Independent message stores and message transport agents | |
| CN115658347A (en) | Data consumption method, device, electronic equipment, storage medium and program product | |
| CN112600943A (en) | Message synchronization method of heterogeneous system in high-concurrency data state | |
| JP3672465B2 (en) | File storage device | |
| CN109688194A (en) | A kind of resource downloading method | |
| CN111061719A (en) | Data collection method, device, equipment and storage medium | |
| US20030126191A1 (en) | System and method for distributing process-related information in a multiple node network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |