CN110213317A - The method, apparatus and storage medium of message storage - Google Patents
The method, apparatus and storage medium of message storage Download PDFInfo
- Publication number
- CN110213317A CN110213317A CN201810791254.8A CN201810791254A CN110213317A CN 110213317 A CN110213317 A CN 110213317A CN 201810791254 A CN201810791254 A CN 201810791254A CN 110213317 A CN110213317 A CN 110213317A
- Authority
- CN
- China
- Prior art keywords
- message
- request
- node
- preset number
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses the method, apparatus and storage medium of a kind of storage of message, belong to Internet technical field.The described method includes: receiving at least one message production request of at least one message source node transmission, the message production request carries first message to be transmitted；The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation reach the first preset number, request is synchronously written to the transmission of at least one second node, and the write-in journal entries of first preset number are subjected to persistence processing, obtain the target write-in journal entries of first preset number；From the target of first preset number write-in journal entries, the target message for obtaining first preset number produces request, stores the first message that the target message production request of first preset number carries.Using the present invention, the treatment effeciency of message-oriented middleware cluster can be improved.
Description
Technical field
The present invention relates to Internet technical field, in particular to a kind of the method, apparatus and storage medium of message storage.
Background technique
With the development of internet technology, need to transmit the message of magnanimity in network, message delivery system, which generally uses, to be divided
Cloth system is handled.
Nervous centralis of the message-oriented middleware cluster as intermodule interacting message in distributed system guarantees that the height of message can
It is one of prerequisite fundamental characteristics by transmission.RabbitMQ cluster is a kind of widely used highly reliable message-oriented middleware,
It forms a cluster by multiple nodes.Message to be transmitted can be sent to RabbitMQ cluster by message source node,
In RabbitMQ cluster, message needs to store portion respectively in all nodes, passes through GM (Guaranteed between node
Multicast, reliable multicast) algorithm carrys out synchronization message, guarantee the highly reliable strongly consistent of message.The process of message storage may is that
All nodes in RabbitMQ cluster form a circular linked list, and first node can receive disappearing for message source node transmission
Breath, and next second node of circular linked list is transmitted the message to, message is after first node sending, successively according to circular linked list
It is transmitted to each second node, final first node will receive the message, show that message has been copied to all second nodes.
But since RabbitMQ cluster needs all second nodes all to store the message of first node sending, just to disappearing
Source node feeding back confirmation message is ceased, the time-consuming of part cluster-based storage message, leads to message-oriented middleware cluster between increasing in the message
Treatment effeciency it is lower.
Summary of the invention
The embodiment of the invention provides the method, apparatus and storage medium of a kind of storage of message, can be improved among message
The treatment effeciency of part cluster.The technical solution is as follows:
On the one hand, a kind of method of message storage is provided, this method is applied to the first node of message-oriented middleware cluster,
This method comprises:
At least one message production request of at least one message source node transmission is received, the message production request carries
The first message of available for transmission；
The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation
When reaching the first preset number, the write-in journal entries for carrying first preset number are sent at least one second node
Be synchronously written request, and the write-in journal entries of first preset number are subjected to persistence processing, obtain described first
Journal entries are written in the target of preset number；
From the target of first preset number write-in journal entries, the target message of first preset number is obtained
Production request stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of method of message storage is provided, this method is applied to the second node of message-oriented middleware, the party
Method includes:
The request that is synchronously written of first node transmission is received, described be synchronously written in request carries the first preset number
Journal entries are written, said write journal entries produce request by corresponding message and generate, and carry in the message production request
The first message of available for transmission；
The write-in journal entries of first preset number are subjected to persistence processing, obtain first preset number
Journal entries are written in target；
From the target of first preset number write-in journal entries, the target message of first preset number is obtained
Production request stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of device of message storage is provided, which is the first node of message-oriented middleware cluster, the dress
It sets and includes:
Receiving module, it is described to disappear for receiving at least one message production request of at least one message source node transmission
Breath production request carries first message to be transmitted；
Log persistence module, for generating corresponding write-in journal entries based on the production request of each message and caching,
When the write-in journal entries of generation reach the first preset number, it is pre- that described first is carried to the transmission of at least one second node
If the write-in journal entries of number are synchronously written request, and the write-in journal entries of first preset number are carried out persistently
Change processing obtains the target write-in journal entries of first preset number；
Memory module, for it is default to obtain described first from the target of first preset number write-in journal entries
The target message of number produces request, stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of device of message storage is provided, which is the second node of message-oriented middleware, the device packet
It includes:
Receiving module, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request is carried
The write-in journal entries of first preset number, said write journal entries produce request by corresponding message and generate, the message
First message to be transmitted is carried in production request；
Log persistence module is obtained for the write-in journal entries of first preset number to be carried out persistence processing
Journal entries are written in target to first preset number；
Memory module, for it is default to obtain described first from the target of first preset number write-in journal entries
The target message of number produces request, stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of server is provided, which includes processor and memory, is stored in the memory
At least one instruction, at least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, institute
State code set or instruction set loaded by the processor and executed with realize one side face message storage method；Or it is above-mentioned
The method of the message storage of another aspect.
On the one hand, provide a kind of computer readable storage medium, be stored in the storage medium at least one instruction,
At least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or refer to
The method for enabling collection be loaded by the processor and executed to realize the message storage in one side face；Or above-mentioned another aspect disappears
Cease the method for storage.
Technical solution provided in an embodiment of the present invention has the benefit that
In the embodiment of the present invention, first node and second node in message-oriented middleware cluster can be to write-in log items
Mesh deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message, subtracts
Few number of processing, can reduce the time-consuming of message-oriented middleware cluster, improve the treatment effeciency of message-oriented middleware.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is a kind of implementation environment figure provided in an embodiment of the present invention；
Fig. 2 is a kind of message-oriented middleware cluster schematic diagram provided in an embodiment of the present invention；
Fig. 3 is a kind of method flow diagram of message storage provided in an embodiment of the present invention；
Fig. 4 is the method flow diagram that a kind of message provided in an embodiment of the present invention is deleted；
Fig. 5 is a kind of message-oriented middleware clustering performance contrast schematic diagram provided in an embodiment of the present invention；
Fig. 6 is a kind of schematic device of message storage provided in an embodiment of the present invention；
Fig. 7 is a kind of schematic device of message storage provided in an embodiment of the present invention；
Fig. 8 is a kind of structural schematic diagram of server provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
Fig. 1 is a kind of implementation environment figure provided in an embodiment of the present invention.The implementation environment may include multiple electronic equipments
101, for providing the message-oriented middleware cluster 102 of service for multiple electronic equipment 101.Multiple electronic equipments 101 pass through nothing
Line or cable network and message-oriented middleware cluster 102 connect, and multiple electronic equipment 101 can be to be able to access that among message
Computer equipment, intelligent terminal of part cluster 102 etc..One electronic equipment 101 can pass through message-oriented middleware cluster 102 to its
Its electronic equipment 101 sends message, at this point, the electronic equipment 101 for sending message is message source node, receives the electronics of message
Equipment 101 is message destination node.It can store the message of message source node transmission in message-oriented middleware cluster 102, and
And can be in message-oriented middleware cluster 102 and message is stored in such a way that more copies store, it is lost to avoid single copy
Mistake or wrong the case where then data can not be recovered, improve the reliability of message transmission.
The treatment effeciency of message-oriented middleware cluster can be measured using many indexes, such as throughput of system, QPS
(Queries Per Second, query rate per second), number of concurrent, response time etc..QPS can be generally used for one service of instruction
Device number of request handled at the appointed time, therefore, in the embodiment of the present invention by taking QPS as an example, for measuring message-oriented middleware
The treatment effeciency of cluster.In message transmitting procedure, message-oriented middleware cluster, which carries out more copy storages to message, can be consumed centainly
Time, and how to reduce the time of message-oriented middleware cluster consumption to improve the QPS of message-oriented middleware cluster, be in message
Between part cluster a core technology point.The embodiment of the invention provides a kind of methods of message storage, it is possible to reduce in message
Between part cluster consume time.
This method can realize by message-oriented middleware cluster, the method flow diagram of message storage as shown in connection with fig. 3, the party
The process flow of method may include following step:
301, first node receives at least one message production request of at least one message source node transmission.
When message source node needs message being transferred to message destination node, first message to be transmitted can be added
Into message production request, message production request then is sent to first node.During the present invention is implemented, message-oriented middleware cluster
In message production process can refer to more copy storing process of message, also i.e. by message storage to message-oriented middleware cluster
In each node.Message production request can wrap containing the instruction (command) for executing write operation, can serve to indicate that first
Node stores message entrained in message production request.
Of course, it is possible to there is more than one message source node to send message production request, a message source section to first node
Point can also send the production request of more than one message.On this basis, first node can receive multiple message productions and ask
It asks.
302, first node is based on the corresponding write-in journal entries of each message production request generation and caches, when generation
When write-in journal entries reach the first preset number, the write-in journal entries of the first preset number are subjected to persistence processing, are obtained
Journal entries are written in target to the first preset number.
Message production request can be added to preset by first node whenever receiving a message production request
In the frame for instructing log, a write-in journal entries are generated, and the write-in journal entries can be cached.Only to writing
When entering journal entries and being cached, the write-in journal entries that first node caches after restarting will be deleted and possibly can not give for change,
And treated that write-in journal entries will not be deleted after first node is restarted for persistence, guarantees the write-in day of persistence
Will entry is not lost, and the reliability of message storage is improved.Persistence processing may include write-in disk or store in the database
The specific processing of persistence is not construed as limiting Deng, the embodiment of the present invention.
In general, one write-in journal entries of the every generation of first node, it is lasting to carry out to the write-in journal entries
Change processing.But persistence processing is carried out to write-in journal entries, the time for needing to consume is longer, in the biggish feelings of request amount
Under condition, persistence processing is carried out if one write-in journal entries of every generation, the QPS of message-oriented middleware cluster can compare
It is low.Therefore, the embodiment of the invention provides a kind of method for carrying out persistence processing to write-in journal entries, batch carries out lasting
Change.That is, when the write-in journal entries of generation reach the first preset number, just by the write-in log item of first preset number
Mesh carries out persistence processing, for example, just writing this 100 write-in journal entries when storing 100 write-in journal entries in caching
Enter disk.After the completion of persistence processing, corresponding write-in journal entries in caching can be deleted.
It is found by practice, carry out the time-consuming of primary batch persistence and one is written the time-consuming of journal entries persistence
It compares, does not grow proportionately, and time-consuming gap is little.The method that batch provided in an embodiment of the present invention carries out persistence subtracts
Lack persistence number of processing, can effectively improve the QPS of message-oriented middleware cluster, and then improve message-oriented middleware cluster
Treatment effeciency.
In a kind of possible embodiment, for different request amounts, the first preset number of above-mentioned batch persistence
It can be different.The corresponding relationship of request amount Yu the first preset number can be previously provided in first node.First node can be with
Real-time detection current request amount, or current request amount can be periodically detected.Then, first node can be in request amount and the
In the corresponding relationship of one preset number, corresponding first preset number of current request amount is determined, and then be based on first present count
Mesh executes the processing of above-mentioned steps 302, and details are not described herein again.The number of request amount is not fixed, and first node is pre-set right
It should be related to the corresponding relationship that can also be request amount range Yu the first preset number, be not construed as limiting herein.
For example, corresponding first preset number can also be larger when request amount is larger, guarantee the different request amounts the case where
Under, persistence number of processing is roughly the same, maintains higher QPS on the basis of improving the QPS of message-oriented middleware cluster.
303, first node obtains the target of the first preset number from the target of the first preset number write-in journal entries
Message production request, the first message that the target message production request of the first preset number of storage carries.
Step 302 and step 303 can be asynchronous implement namely step 302 and step 303 can be real by different processes
Existing, step 302 can carry out batch persistence processing next time, without waiting for walking when completing primary batch persistence
Rapid 303 release process.In step 303, the write-in log persistently handled has been completed in the available step 302 of first node
Entry executes the instruction of the write operation in corresponding message production request, corresponding first message to be transmitted is deposited
Storage.Storage processing is also possible to persistence processing, the first message to be transmitted so as to persistent storage.
304, when the write-in journal entries of generation reach the first preset number, first node is at least one second node
Transmission is synchronously written request.
In above-mentioned steps 302, when the write-in journal entries of caching reach the first preset number, first node can be closed
And journal entries are written in first preset number, generate one and are synchronously written request, then can to its corresponding all the
Two nodes send this and are synchronously written request, so that second node stores message.This, which is synchronously written request, can carry above-mentioned
The write-in journal entries of one preset number.First node can be synchronously written simultaneously to its corresponding all second node transmission and ask
It asks, is also possible to each second node broadcast synchronization write request, to first node to second node in the embodiment of the present invention
The concrete mode that transmission is synchronously written request is not construed as limiting.
In a kind of possible embodiment, first node can be requested synchronization removal based at least one second node
Processing progress, determine the number of at least one second node untreated synchronization removal request.When untreated synchronization removal
When the number of request reaches the second number upper limit, first node, which suspends at least one second node, sends synchronization removal request,
Until the number of untreated synchronization removal request is less than the second number upper limit.
First node can recorde be sent to each second node be synchronously written request, when receiving any second node
Confirmation message when, by it is corresponding be synchronously written request marks be have confirmed that (committed), in turn, first node can unite
Count the untreated number for being synchronously written request.
When the untreated number for being synchronously written request of any second node reaches the first number upper limit (such as 400)
When, first node, which can suspend to second node transmission, is synchronously written request, until the untreated number for being synchronously written request
Mesh just continues transmission and is synchronously written request less than the first number upper limit.By above-mentioned processing, first node is sent to second node
After one is synchronously written request, second node can be not to wait for and send the confirmation message for being synchronously written request, that is, under can be transmitted
One is synchronously written request, improves the treatment effeciency of message-oriented middleware cluster.For example, first node can pass through pipeline
(pipeline) sends multiple RPC (Remote Procedure Call, remote procedure call) requests, and can count untreated
RPC request number.
After first node is synchronously written request to the transmission of at least one second node, corresponding second node can be received together
Walk write request.
305, when second node, which receives, is synchronously written request, the write-in journal entries of the first preset number are held
Longization processing obtains the target write-in journal entries of the first preset number.
Any second node receives when being synchronously written request, and available this is synchronously written the first present count in request
Journal entries are written in purpose, and the write-in journal entries of first preset number are carried out the processing of batch persistence, detailed process
With first node step 302 persistence treatment process similarly, details are not described herein again.
306, second node is synchronously written the corresponding write-in confirmation message of request to first node transmission.
When second node completion is lasting to the batch of the write-in journal entries for the first preset number being synchronously written in request
When changing processing, write-in confirmation message can be sent to first node.This process withouts waiting for second node completely to be transmitted
Message store and complete, it is only necessary to the write-in journal entries batch persistence of the first preset number is handled and is completed, is reduced
The interaction of first node and second node is time-consuming, improves the QPS of message-oriented middleware cluster, and then improve message-oriented middleware cluster
Treatment effeciency.
307, second node obtains the target of the first preset number from the target of the first preset number write-in journal entries
Message production request, the first message that the target message production request of the first preset number of storage carries.
When second node completes the persistence processing to the first preset number in step 305, second node can be obtained
Persistence is taken treated target write-in log, executes the instruction of the write operation in corresponding message production request, will correspond to
First message to be transmitted stored.Detailed process and the message storing process of first node in step 303 similarly, this
Place repeats no more.Since persistence treated target write-in log will not lose since second node is restarted, when the
When restarting after two nodes break downs, message can be continued based on the target write-in log for having completed persistence processing and deposited
Storage, guarantees that the message having been acknowledged is not lost.
308, first node sends message to the message producer when the write-in confirmation message received meets preset rules
Produce success message.
Request is produced for the message that any message source node is sent, by the processing of step 301-307, first node can
To receive the write-in confirmation message (confirm) of at least one second node transmission, and then the write-in received can be counted
The number of confirmation message.When the write-in confirmation message received meets preset rules, first node can be to the message source section
Point sends message and produces success message, which can serve to indicate that the number of write-in confirmation message is more than preset threshold.
Preset rules can be write-in confirmation message number be more than message-oriented middleware cluster node total number half, alternatively, may be used also
It is more than 2/3rds of the node total number of message-oriented middleware cluster with the number for being write-in confirmation message.The embodiment of the present invention is to tool
The preset rules of body are not construed as limiting.For example, preset rules are section of the number more than message-oriented middleware cluster that confirmation message is written
When the half of point sum, if the node total number of message-oriented middleware cluster is 3, when first node receives 2 write-in confirmations
When thering are 2 node processings to complete in message, namely 3 nodes of composition message-oriented middleware cluster, it can be sent out to message source node
Message is sent to produce success message, needing not wait for 3 nodes, all processing is completed.
It can be seen that first node, which withouts waiting for each second node, sends write-in confirmation message, as long as being more than pre-
If the second node of threshold value has sent write-in confirmation message, disappear for causing second node to fail to timely feedback write-in confirmation
The factor of breath, such as network delay have certain tolerance, improve the reliability of message-oriented middleware cluster.
In the above process after the completion of message production, message destination node can pull message from first node, pull message
The detailed process embodiment of the present invention do not introduce.After message destination node is disposed to message, it can enable among message
Part cluster deletes the message.The process flow of the method flow diagram that message as shown in connection with fig. 4 is deleted, the method that message is deleted can
To include the following steps:
401, first node receives at least one message removal request of at least one message destination node transmission.
After message destination node is disposed to second message, it can send to first node and disappear with to be deleted second
The corresponding message removal request of manner of breathing, which can carry the mark of second message to be deleted, for example, disappearing
Breath removal request can be an ACK (acknowledgement, confirmation) message, is used to indicate and has handled to second message
Finish.
Of course, it is possible to there is more than one message destination node to send message removal request, a message mesh to first node
Node can also send more than one message removal request.On this basis, first node can receive multiple message and delete
Except request.
402, first node is based on the corresponding deletion journal entries of each message removal request generation and caches, when generation
When deletion journal entries reach third preset number, the deletion journal entries of third preset number are subjected to persistence processing, are obtained
Target to third preset number deletes journal entries.
Similarly, details are not described herein again for the processing of step 402 and above-mentioned steps 302.Delete the third present count of journal entries
Mesh and the first preset number of write-in journal entries may be the same or different, and technical staff, which can according to need, to be set
It sets.
403, first node is deleted in journal entries from the target of third preset number, obtains the target of third preset number
Message removal request deletes the second message of the target message removal request of third preset number.
Similarly, difference is for the processing of step 403 and above-mentioned steps 303, and can be for carrying in deletion journal entries is deleted
Except the instruction of operation, first node can second message progress of the instruction based on the delete operation to persistence processing is completed
It deletes.
404, when the deletion journal entries of generation reach third preset number, first node is at least one second node
Send synchronization removal request.
Similarly, details are not described herein again for the processing of step 404 and above-mentioned steps 304.
In a kind of possible embodiment, first node can be requested synchronization removal based at least one second node
Processing progress, determine the number of at least one second node untreated synchronization removal request.When untreated synchronization removal
When the number of request reaches the second number upper limit, suspends at least one second node and send synchronization removal request, until not locating
The number of the synchronization removal request of reason is less than the second number upper limit.
After first node sends synchronization removal request at least one second node, corresponding second node can be received together
Walk removal request.
405, when second node receives synchronization removal request, the deletion journal entries of third preset number are held
Longization processing, the target for obtaining third preset number delete journal entries.
406, second node sends synchronization removal to first node and requests corresponding deletion confirmation message.
407, second node is deleted in journal entries from the target of third preset number, obtains the target of third preset number
Message removal request deletes the second message of the target message removal request of third preset number.
Similarly, details are not described herein again for the processing of step 405-407 and above-mentioned steps 305-307.
Any second node is sent after synchronization removal requests corresponding deletions confirmation message to first node, first node with
Any second node can delete corresponding deletion journal entries, alternatively, can be by corresponding deletion journal entries
It stores in other storage locations, to recall to the deletion journal entries, deletion of the embodiment of the present invention to having confirmed that
The specific processing of journal entries is not construed as limiting.
If first node, which needs to send to message destination node, deletes confirmation message, can also be based on and step 308
Method similarly is handled, and details are not described herein again.
Destination node is consumed after pulling message to first node, when completing to the Message Processing, by active to first
Node sends the mode of message removal request, explicitly informs that the first node message has handled success.In this way, among message
Part cluster can just delete the message by above-mentioned steps 401-407, if the message of consumption destination node is lost,
Message can be pulled to first node again by then consuming destination node, it is ensured that before consumption destination node processing is completed, be disappeared
Breath will not be lost, and the reliability of message-oriented middleware cluster is improved.
Illustratively, the message-oriented middleware cluster introduced in the above process can be CMQ (Cloud Message Queue,
Cloud message queue), CMQ is a kind of highly reliable high performance financial level message-oriented middleware cluster, passes through raft between the node of CMQ
(raft) algorithm guarantees highly reliable strongly consistent, it is ensured that the reliability of message storage.Message-oriented middleware cluster signal as shown in Figure 2
Figure, in a message-oriented middleware cluster can by first node (leader, leader node) and second node (follower, with
With node) it constitutes, consistency can be set on first node and second node and guaranteed module, instruction log (command
) and state machine (state machine) log.First node can be used as the leader node of at least one second node, and instruction is extremely
A few second node carries out storage or delete operation.Message source node, can be to message-oriented middleware collection as the message producer
The first node of group sends message production request.Then, first node can be generated message production and request corresponding write-in
(entry) journal entries, and guarantee that journal entries storage will be written into instruction log in module by consistency, also, may be used also
To guarantee that module will be written journal entries generation and be synchronously written request by consistency, it is sent to each second node, realizes step
It is rapid to carry out above-mentioned steps 301-302,304,.Second node receives when being synchronously written request, can guarantee mould by consistency
Journal entries storage will be written into instruction log in block, realize step 305-306.Write-in journal entries can be applied in first segment
On the state machine of point and second node, the message in the available write-in journal entries of state machine produces request, by message source section
Disk is written in the message that point is sent, and updates the ApplyIndex (application parameter) stored in state machine, realizes step 303,307.
Finally, first node can feed back write-in confirmation message to message source node, step 308 is realized.
Message destination node can pull message to first node as message consumer, to carry out Message Processing.
The deletion process of CMQ and it is above-mentioned similarly, details are not described herein again.Message-oriented middleware clustering performance contrast schematic diagram as shown in Figure 5,
The data volume size of every message of CMQ and RabbitMQ processing is all 1KB, CMQ method based on the embodiment of the present invention
It is handled, the QPS of CMQ is 4 times of RabbitMQ or so, and performance is far more than RabbitMQ.
In the embodiment of the present invention, first node and second node in message-oriented middleware cluster can be to write-in log items
Mesh deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message, subtracts
Few number of processing, can reduce the time-consuming of message-oriented middleware cluster, improve the treatment effeciency of message-oriented middleware.Also, first
Node can send message to the message producer and produce successfully when the write-in confirmation message satisfaction received meets preset rules
Message withouts waiting for all second nodes and all sends write-in confirmation message, in the same of the time-consuming for reducing message-oriented middleware cluster
When, the factor for causing second node to fail to timely feedback write-in confirmation message can be improved and disappear with certain tolerance
Cease the reliability of middleware cluster.
Based on the same technical idea, the embodiment of the invention also provides a kind of device of message storage, which can be with
It is the first node in above-described embodiment.As shown in fig. 6, the device includes:
Receiving module 610, it is described for receiving at least one message production request of at least one message source node transmission
Message production request carries first message to be transmitted；
Log persistence module 620, for generating corresponding write-in journal entries based on the production request of each message and delaying
It deposits, when the write-in journal entries of generation reach the first preset number, carries described the to the transmission of at least one second node
The write-in journal entries of one preset number are synchronously written request, and the write-in journal entries of first preset number are carried out
Persistence processing obtains the target write-in journal entries of first preset number；
Memory module 630, for it is pre- to obtain described first from the target of first preset number write-in journal entries
If the target message of number produces request, the target message production for storing first preset number requests carry first to disappear
Breath.
Optionally, described device further include:
Determining module 640, for, to the processing progress for being synchronously written request, being determined based at least one described second node
The untreated number for being synchronously written request of at least one second node；
The log persistence module 620 is also used to reach first when the untreated number for being synchronously written request
When the number upper limit, suspends to the transmission of at least one described second node and be synchronously written request, until the untreated synchronous write
The number for entering request is less than the first number upper limit.
Optionally, described device further include:
Sending module 650 is used for for any message source node, when the write-in confirmation message received meets preset rules
When, Xiang Suoshu message source node sends message and produces success message, wherein said write confirmation message is used to indicate second node
Persistence processing is carried out to the message source node corresponding target write-in journal entries to complete.
Optionally, the receiving module 610 is also used to receive at least one that at least one message destination node is sent and disappears
Removal request is ceased, the message removal request is corresponding with second message to be deleted；
The log persistence module 620 is also used to generate corresponding deletion log item based on each message removal request
Mesh simultaneously caches, and when the deletion journal entries of generation reach third preset number, takes to the transmission of at least one described second node
The synchronization removals of deletion journal entries with the third preset number is requested, and by the deletion day of the third preset number
Will entry carries out persistence processing, and the target for obtaining the third preset number deletes journal entries；
Described device further include:
It is pre- to obtain the third for deleting in journal entries from the target of the third preset number for removing module 660
If the target message removal request of number deletes the second message of the target message removal request of the third preset number.
Optionally, the determining module 640 is also used to request synchronization removal based at least one described second node
Processing progress determines the number of the untreated synchronization removal request of at least one described second node；
The log persistence module 620 is also used to reach second when the number of the untreated synchronization removal request
When the number upper limit, suspends at least one described second node and send synchronization removal request, until untreated synchronize is deleted
Except the number of request is less than the second number upper limit.
Based on the same technical idea, the embodiment of the invention also provides a kind of device of message storage, which can be with
It is the second node in above-described embodiment.As shown in fig. 7, the device includes:
Receiving module 710, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request is carried
There are the write-in journal entries of the first preset number, said write journal entries produce request by corresponding message and generate, described to disappear
First message to be transmitted is carried in breath production request；
Log persistence module 720, for the write-in journal entries of first preset number to be carried out persistence processing,
Obtain the target write-in journal entries of first preset number；
Memory module 730, for it is pre- to obtain described first from the target of first preset number write-in journal entries
If the target message of number produces request, the target message production for storing first preset number requests carry first to disappear
Breath.
Optionally, described device further include:
Sending module 740, for being synchronously written the corresponding write-in confirmation message of request described in sending to the first node.
Optionally, the receiving module 710 is also used to receive the synchronization removal request that the first node is sent, described
The deletion journal entries of third preset number are carried in synchronization removal request；
The log persistence module 720, it is lasting to be also used to carry out the deletion journal entries of the third preset number
Change processing, the target for obtaining the third preset number delete journal entries；
Described device further include:
It is pre- to obtain the third for deleting in journal entries from the target of the third preset number for removing module 750
If the target message removal request of number deletes the second message of the target message removal request of the third preset number.
Optionally, the sending module 740, is also used to:
The synchronization removal, which is sent, to the first node requests corresponding deletion confirmation message.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
In foregoing invention embodiment, first node and second node in message-oriented middleware cluster can be to write-in logs
Entry deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message,
Number of processing is reduced, the time-consuming of message-oriented middleware cluster can be reduced, improve the treatment effeciency of message-oriented middleware.
It should be understood that the device of message provided by the above embodiment storage is when message stores, only with above-mentioned each function
Can module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different functions
Module is completed, i.e., the internal structure of first node or second node is divided into different functional modules, to complete above description
All or part of function.In addition, the device of message storage provided by the above embodiment and the embodiment of the method for message storage
Belong to same design, specific implementation process is detailed in embodiment of the method, and which is not described herein again.
Fig. 8 is a kind of structural schematic diagram of server provided in an embodiment of the present invention, which can be because of configuration or property
Energy is different and generates bigger difference, may include one or more processors (central processing
Units, CPU) 801 and one or more memory 802, wherein at least one finger is stored in the memory 802
It enables, at least one instruction is loaded by the processor 801 and executed to realize disappearing for above-mentioned first node or second node
Cease the method and step of storage.
In the exemplary embodiment, a kind of computer readable storage medium is additionally provided, the memory for example including instruction,
The method that above-metioned instruction can be executed by the processor in server to complete above-mentioned message storage.For example, described computer-readable
Storage medium can be ROM, random access memory (RAM), CD-ROM, tape, floppy disk and optical data storage devices etc..
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware
It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (15)
1. a kind of method of message storage, which is characterized in that the method is applied to the first node of message-oriented middleware cluster, institute
The method of stating includes:
Receive at least one message production request of at least one message source node transmission, message production request carry to
The first message of transmission；
The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation reach
When the first preset number, the same of the write-in journal entries for carrying first preset number is sent at least one second node
Write request is walked, and the write-in journal entries of first preset number are subjected to persistence processing, it is default to obtain described first
Journal entries are written in the target of number；
From the target of first preset number write-in journal entries, the target message production of first preset number is obtained
Request stores the first message that the target message production request of first preset number carries.
2. the method according to claim 1, wherein the method also includes:
Based at least one described second node to the processing progress for being synchronously written request, at least one described second node is determined
The untreated number for being synchronously written request；
When the untreated number for being synchronously written request reaches the first number upper limit, suspend to it is described at least one second
Node transmission is synchronously written request, until the untreated number for being synchronously written request is less than the first number upper limit.
3. the method according to claim 1, wherein the method also includes:
For any message source node, when the write-in confirmation message received meets preset rules, Xiang Suoshu message source node
It sends message and produces success message, wherein said write confirmation message is used to indicate second node to the message source node pair
The target write-in journal entries answered carry out persistence processing and complete.
4. the method according to claim 1, wherein the method also includes:
Receive at least one message destination node transmission at least one message removal request, the message removal request with wait delete
The second message removed is corresponding；
Corresponding deletion journal entries are generated based on each message removal request and are cached, when the deletion journal entries of generation reach
When third preset number, the deletion journal entries for carrying the third preset number are sent at least one described second node
Synchronization removal request, and by the deletion journal entries of the third preset number carry out persistence processing, obtain the third
The target of preset number deletes journal entries；
It is deleted in journal entries from the target of the third preset number, the target message for obtaining the third preset number is deleted
Request, deletes the second message of the target message removal request of the third preset number.
5. according to the method described in claim 4, it is characterized in that, the method also includes:
Based on the processing progress that at least one described second node requests synchronization removal, at least one described second node is determined
The number of untreated synchronization removal request；
When the number of the untreated synchronization removal request reaches the second number upper limit, suspend to it is described at least one second
Node sends synchronization removal request, until the number of the untreated synchronization removal request is less than the second number upper limit.
6. a kind of method of message storage, which is characterized in that the method is applied to the second node of message-oriented middleware, the side
Method includes:
The request that is synchronously written of first node transmission is received, it is described to be synchronously written the write-in that the first preset number is carried in request
Journal entries, said write journal entries produce request by corresponding message and generate, carried in message production request to
The first message of transmission；
The write-in journal entries of first preset number are subjected to persistence processing, obtain the target of first preset number
Journal entries are written；
From the target of first preset number write-in journal entries, the target message production of first preset number is obtained
Request stores the first message that the target message production request of first preset number carries.
7. according to the method described in claim 6, it is characterized in that, the write-in journal entries by first preset number
After progress persistence processing, further includes:
Corresponding write-in confirmation message is requested to being synchronously written described in first node transmission.
8. according to the method described in claim 6, it is characterized in that, the method also includes:
The synchronization removal request that the first node is sent is received, carries third preset number in the synchronization removal request
Delete journal entries；
The deletion journal entries of the third preset number are subjected to persistence processing, obtain the target of the third preset number
Delete journal entries；
It is deleted in journal entries from the target of the third preset number, the target message for obtaining the third preset number is deleted
Request, deletes the second message of the target message removal request of the third preset number.
9. according to the method described in claim 8, it is characterized in that, the deletion journal entries by the third preset number
After progress persistence processing, further includes:
The synchronization removal, which is sent, to the first node requests corresponding deletion confirmation message.
10. a kind of device of message storage, which is characterized in that described device is the first node of message-oriented middleware cluster, described
Device includes:
Receiving module, for receiving at least one message production request of at least one message source node transmission, the message is raw
It produces request and carries first message to be transmitted；
Log persistence module works as life for generating corresponding write-in journal entries based on the production request of each message and caching
At write-in journal entries reach the first preset number when, at least one second node transmission carry first present count
Purpose write-in journal entries are synchronously written request, and the write-in journal entries of first preset number are carried out at persistence
Reason obtains the target write-in journal entries of first preset number；
Memory module, for obtaining first preset number from the target of first preset number write-in journal entries
Target message produce request, store the first message that the target message production request of first preset number carries.
11. device according to claim 10, which is characterized in that described device further include:
Determining module, for, to the processing progress for being synchronously written request, determination to be described extremely based at least one described second node
Few untreated number for being synchronously written request of a second node；
The log persistence module is also used to reach the first number upper limit when the untreated number for being synchronously written request
When, suspend to the transmission of described at least one second node and be synchronously written request, untreated is synchronously written request until described
Number is less than the first number upper limit.
12. device according to claim 10, which is characterized in that described device further include:
Sending module is used for for any message source node, when the write-in confirmation message received meets preset rules, to institute
It states message source node and sends message production success message, wherein said write confirmation message is used to indicate second node to described
The corresponding target write-in journal entries of message source node carry out persistence processing and complete.
13. a kind of device of message storage, which is characterized in that described device is the second node of message-oriented middleware, described device
Include:
Receiving module, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request carries first
The write-in journal entries of preset number, said write journal entries produce request by corresponding message and generate, the message production
First message to be transmitted is carried in request；
Log persistence module obtains institute for the write-in journal entries of first preset number to be carried out persistence processing
State the target write-in journal entries of the first preset number；
Memory module, for obtaining first preset number from the target of first preset number write-in journal entries
Target message produce request, store the first message that the target message production request of first preset number carries.
14. a kind of server, which is characterized in that the server includes processor and memory, is stored in the memory
At least one instruction, at least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, institute
Code set or instruction set is stated to be loaded by the processor and executed to realize message storage as claimed in claim 1 to 5
Method；Or the method for the message storage as described in claim 6 to 9 is any.
15. a kind of computer readable storage medium, which is characterized in that be stored at least one instruction, extremely in the storage medium
A few Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or instruction
The method that collection is loaded by the processor and executed to realize message storage as claimed in claim 1 to 5；As right is wanted
The method for asking 6 to 9 any message storages.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810791254.8A CN110213317B (en) | 2018-07-18 | 2018-07-18 | Message storage method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810791254.8A CN110213317B (en) | 2018-07-18 | 2018-07-18 | Message storage method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110213317A true CN110213317A (en) | 2019-09-06 |
| CN110213317B CN110213317B (en) | 2021-10-29 |
Family
ID=67779788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810791254.8A Active CN110213317B (en) | 2018-07-18 | 2018-07-18 | Message storage method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110213317B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111880949A (en) * | 2020-07-28 | 2020-11-03 | 北京奇艺世纪科技有限公司 | Method, system, device, server and terminal for processing service information |
| CN112732493A (en) * | 2021-03-30 | 2021-04-30 | 恒生电子股份有限公司 | Method and device for newly adding node, node of distributed system and storage medium |
| CN112822227A (en) * | 2019-11-15 | 2021-05-18 | 北京金山云网络技术有限公司 | Data synchronization method, device, equipment and storage medium of distributed storage system |
| CN113839923A (en) * | 2021-08-28 | 2021-12-24 | 西安交通大学 | Multi-node-oriented high-performance processing method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7085803B1 (en) * | 1996-11-05 | 2006-08-01 | Oracle International Corporation | Platform-independent programmable batch processing engine |
| CN101707633A (en) * | 2009-11-27 | 2010-05-12 | 山东中创软件商用中间件股份有限公司 | Message-oriented middleware persistent message storing method based on file system |
| CN103488690A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Data integrating system and data integrating method |
| CN103607476A (en) * | 2013-12-05 | 2014-02-26 | 上海普坤信息科技有限公司 | System and method for distributed multi-stage persistence of huge number of messages |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN105978968A (en) * | 2016-05-11 | 2016-09-28 | 山东合天智汇信息技术有限公司 | Real-time transmission processing method, server and system of mass data |
| US20160335287A1 (en) * | 2015-05-14 | 2016-11-17 | Alibaba Group Holding Limited | Stream computing system and method |
| CN106598762A (en) * | 2016-12-29 | 2017-04-26 | 上海理想信息产业（集团）有限公司 | Message synchronization method and system |
-
2018
- 2018-07-18 CN CN201810791254.8A patent/CN110213317B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7085803B1 (en) * | 1996-11-05 | 2006-08-01 | Oracle International Corporation | Platform-independent programmable batch processing engine |
| CN101707633A (en) * | 2009-11-27 | 2010-05-12 | 山东中创软件商用中间件股份有限公司 | Message-oriented middleware persistent message storing method based on file system |
| CN103488690A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Data integrating system and data integrating method |
| CN103607476A (en) * | 2013-12-05 | 2014-02-26 | 上海普坤信息科技有限公司 | System and method for distributed multi-stage persistence of huge number of messages |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| US20160335287A1 (en) * | 2015-05-14 | 2016-11-17 | Alibaba Group Holding Limited | Stream computing system and method |
| CN105262831A (en) * | 2015-10-30 | 2016-01-20 | 北京奇艺世纪科技有限公司 | Method, device, and synchronizing system for synchronizing data among memory systems |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN105978968A (en) * | 2016-05-11 | 2016-09-28 | 山东合天智汇信息技术有限公司 | Real-time transmission processing method, server and system of mass data |
| CN106598762A (en) * | 2016-12-29 | 2017-04-26 | 上海理想信息产业（集团）有限公司 | Message synchronization method and system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822227A (en) * | 2019-11-15 | 2021-05-18 | 北京金山云网络技术有限公司 | Data synchronization method, device, equipment and storage medium of distributed storage system |
| CN112822227B (en) * | 2019-11-15 | 2022-02-25 | 北京金山云网络技术有限公司 | Data synchronization method, device, equipment and storage medium of distributed storage system |
| CN111880949A (en) * | 2020-07-28 | 2020-11-03 | 北京奇艺世纪科技有限公司 | Method, system, device, server and terminal for processing service information |
| CN111880949B (en) * | 2020-07-28 | 2024-03-08 | 北京奇艺世纪科技有限公司 | Service information processing method, system, device, server and terminal |
| CN112732493A (en) * | 2021-03-30 | 2021-04-30 | 恒生电子股份有限公司 | Method and device for newly adding node, node of distributed system and storage medium |
| CN112732493B (en) * | 2021-03-30 | 2021-06-18 | 恒生电子股份有限公司 | Method and device for newly adding node, node of distributed system and storage medium |
| CN113839923A (en) * | 2021-08-28 | 2021-12-24 | 西安交通大学 | Multi-node-oriented high-performance processing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110213317B (en) | 2021-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110213317A (en) | The method, apparatus and storage medium of message storage | |
| CN108595207B (en) | Gray scale publishing method, rule engine, system, terminal and storage medium | |
| US6529921B1 (en) | Dynamic synchronization of tables | |
| CN109117265A (en) | The method, apparatus, equipment and storage medium of schedule job in the cluster | |
| CN112162846B (en) | Transaction processing method, device and computer readable storage medium | |
| WO2022116323A1 (en) | Federated learning scheduling method, device, and system | |
| CN114861911A (en) | Deep learning model training method, device, system, equipment and medium | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| WO2021238301A1 (en) | Application update method, apparatus and device and medium | |
| CN110795235A (en) | Method and system for deep learning and cooperation of mobile web | |
| CN112492026B (en) | Hybrid self-adaptive copy consistency updating method in dynamic cloud storage environment | |
| CN110737510A (en) | Block device management system | |
| CN107465725B (en) | Heterogeneous long transaction processing system and method based on client information control system | |
| CN112463833A (en) | Data set acquisition method, system, device and medium | |
| CN108737485A (en) | For the method and system of the operation of Internet of Things resource | |
| CN111597035A (en) | Simulation engine time advancing method and system based on multiple threads | |
| CN114500352B (en) | Plug-in hot updating system and method for medical internet of things message routing device | |
| CN109728937A (en) | Method and device for updating network state of voice module | |
| CN114610719A (en) | Cross-cluster data processing method and device, electronic equipment and storage medium | |
| CN114201449A (en) | Log monitoring method and device, computer equipment and storage medium | |
| CN113900821A (en) | Task processing method and device, storage medium and electronic equipment | |
| CN106331897B (en) | A kind of list processing method and processing device | |
| CN110851445B (en) | Method for safely storing data based on block chain technology | |
| CN115203219A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115987902A (en) | Service current limiting method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |