CN110213317A - The method, apparatus and storage medium of message storage - Google Patents

The method, apparatus and storage medium of message storage Download PDF

Info

Publication number
CN110213317A
CN110213317A CN201810791254.8A CN201810791254A CN110213317A CN 110213317 A CN110213317 A CN 110213317A CN 201810791254 A CN201810791254 A CN 201810791254A CN 110213317 A CN110213317 A CN 110213317A
Authority
CN
China
Prior art keywords
message
request
node
preset number
write
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810791254.8A
Other languages
Chinese (zh)
Other versions
CN110213317B (en
Inventor
闫二辉
陈云
闫燕飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201810791254.8A priority Critical patent/CN110213317B/en
Publication of CN110213317A publication Critical patent/CN110213317A/en
Application granted granted Critical
Publication of CN110213317B publication Critical patent/CN110213317B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/2866Architectures; Arrangements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Computer And Data Communications (AREA)

Abstract

The invention discloses the method, apparatus and storage medium of a kind of storage of message, belong to Internet technical field.The described method includes: receiving at least one message production request of at least one message source node transmission, the message production request carries first message to be transmitted;The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation reach the first preset number, request is synchronously written to the transmission of at least one second node, and the write-in journal entries of first preset number are subjected to persistence processing, obtain the target write-in journal entries of first preset number;From the target of first preset number write-in journal entries, the target message for obtaining first preset number produces request, stores the first message that the target message production request of first preset number carries.Using the present invention, the treatment effeciency of message-oriented middleware cluster can be improved.

Description

The method, apparatus and storage medium of message storage
Technical field
The present invention relates to Internet technical field, in particular to a kind of the method, apparatus and storage medium of message storage.
Background technique
With the development of internet technology, need to transmit the message of magnanimity in network, message delivery system, which generally uses, to be divided Cloth system is handled.
Nervous centralis of the message-oriented middleware cluster as intermodule interacting message in distributed system guarantees that the height of message can It is one of prerequisite fundamental characteristics by transmission.RabbitMQ cluster is a kind of widely used highly reliable message-oriented middleware, It forms a cluster by multiple nodes.Message to be transmitted can be sent to RabbitMQ cluster by message source node, In RabbitMQ cluster, message needs to store portion respectively in all nodes, passes through GM (Guaranteed between node Multicast, reliable multicast) algorithm carrys out synchronization message, guarantee the highly reliable strongly consistent of message.The process of message storage may is that All nodes in RabbitMQ cluster form a circular linked list, and first node can receive disappearing for message source node transmission Breath, and next second node of circular linked list is transmitted the message to, message is after first node sending, successively according to circular linked list It is transmitted to each second node, final first node will receive the message, show that message has been copied to all second nodes.
But since RabbitMQ cluster needs all second nodes all to store the message of first node sending, just to disappearing Source node feeding back confirmation message is ceased, the time-consuming of part cluster-based storage message, leads to message-oriented middleware cluster between increasing in the message Treatment effeciency it is lower.
Summary of the invention
The embodiment of the invention provides the method, apparatus and storage medium of a kind of storage of message, can be improved among message The treatment effeciency of part cluster.The technical solution is as follows:
On the one hand, a kind of method of message storage is provided, this method is applied to the first node of message-oriented middleware cluster, This method comprises:
At least one message production request of at least one message source node transmission is received, the message production request carries The first message of available for transmission;
The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation When reaching the first preset number, the write-in journal entries for carrying first preset number are sent at least one second node Be synchronously written request, and the write-in journal entries of first preset number are subjected to persistence processing, obtain described first Journal entries are written in the target of preset number;
From the target of first preset number write-in journal entries, the target message of first preset number is obtained Production request stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of method of message storage is provided, this method is applied to the second node of message-oriented middleware, the party Method includes:
The request that is synchronously written of first node transmission is received, described be synchronously written in request carries the first preset number Journal entries are written, said write journal entries produce request by corresponding message and generate, and carry in the message production request The first message of available for transmission;
The write-in journal entries of first preset number are subjected to persistence processing, obtain first preset number Journal entries are written in target;
From the target of first preset number write-in journal entries, the target message of first preset number is obtained Production request stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of device of message storage is provided, which is the first node of message-oriented middleware cluster, the dress It sets and includes:
Receiving module, it is described to disappear for receiving at least one message production request of at least one message source node transmission Breath production request carries first message to be transmitted;
Log persistence module, for generating corresponding write-in journal entries based on the production request of each message and caching, When the write-in journal entries of generation reach the first preset number, it is pre- that described first is carried to the transmission of at least one second node If the write-in journal entries of number are synchronously written request, and the write-in journal entries of first preset number are carried out persistently Change processing obtains the target write-in journal entries of first preset number;
Memory module, for it is default to obtain described first from the target of first preset number write-in journal entries The target message of number produces request, stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of device of message storage is provided, which is the second node of message-oriented middleware, the device packet It includes:
Receiving module, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request is carried The write-in journal entries of first preset number, said write journal entries produce request by corresponding message and generate, the message First message to be transmitted is carried in production request;
Log persistence module is obtained for the write-in journal entries of first preset number to be carried out persistence processing Journal entries are written in target to first preset number;
Memory module, for it is default to obtain described first from the target of first preset number write-in journal entries The target message of number produces request, stores the first message that the target message production request of first preset number carries.
On the one hand, a kind of server is provided, which includes processor and memory, is stored in the memory At least one instruction, at least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, institute State code set or instruction set loaded by the processor and executed with realize one side face message storage method;Or it is above-mentioned The method of the message storage of another aspect.
On the one hand, provide a kind of computer readable storage medium, be stored in the storage medium at least one instruction, At least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or refer to The method for enabling collection be loaded by the processor and executed to realize the message storage in one side face;Or above-mentioned another aspect disappears Cease the method for storage.
Technical solution provided in an embodiment of the present invention has the benefit that
In the embodiment of the present invention, first node and second node in message-oriented middleware cluster can be to write-in log items Mesh deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message, subtracts Few number of processing, can reduce the time-consuming of message-oriented middleware cluster, improve the treatment effeciency of message-oriented middleware.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other Attached drawing.
Fig. 1 is a kind of implementation environment figure provided in an embodiment of the present invention;
Fig. 2 is a kind of message-oriented middleware cluster schematic diagram provided in an embodiment of the present invention;
Fig. 3 is a kind of method flow diagram of message storage provided in an embodiment of the present invention;
Fig. 4 is the method flow diagram that a kind of message provided in an embodiment of the present invention is deleted;
Fig. 5 is a kind of message-oriented middleware clustering performance contrast schematic diagram provided in an embodiment of the present invention;
Fig. 6 is a kind of schematic device of message storage provided in an embodiment of the present invention;
Fig. 7 is a kind of schematic device of message storage provided in an embodiment of the present invention;
Fig. 8 is a kind of structural schematic diagram of server provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention Formula is described in further detail.
Fig. 1 is a kind of implementation environment figure provided in an embodiment of the present invention.The implementation environment may include multiple electronic equipments 101, for providing the message-oriented middleware cluster 102 of service for multiple electronic equipment 101.Multiple electronic equipments 101 pass through nothing Line or cable network and message-oriented middleware cluster 102 connect, and multiple electronic equipment 101 can be to be able to access that among message Computer equipment, intelligent terminal of part cluster 102 etc..One electronic equipment 101 can pass through message-oriented middleware cluster 102 to its Its electronic equipment 101 sends message, at this point, the electronic equipment 101 for sending message is message source node, receives the electronics of message Equipment 101 is message destination node.It can store the message of message source node transmission in message-oriented middleware cluster 102, and And can be in message-oriented middleware cluster 102 and message is stored in such a way that more copies store, it is lost to avoid single copy Mistake or wrong the case where then data can not be recovered, improve the reliability of message transmission.
The treatment effeciency of message-oriented middleware cluster can be measured using many indexes, such as throughput of system, QPS (Queries Per Second, query rate per second), number of concurrent, response time etc..QPS can be generally used for one service of instruction Device number of request handled at the appointed time, therefore, in the embodiment of the present invention by taking QPS as an example, for measuring message-oriented middleware The treatment effeciency of cluster.In message transmitting procedure, message-oriented middleware cluster, which carries out more copy storages to message, can be consumed centainly Time, and how to reduce the time of message-oriented middleware cluster consumption to improve the QPS of message-oriented middleware cluster, be in message Between part cluster a core technology point.The embodiment of the invention provides a kind of methods of message storage, it is possible to reduce in message Between part cluster consume time.
This method can realize by message-oriented middleware cluster, the method flow diagram of message storage as shown in connection with fig. 3, the party The process flow of method may include following step:
301, first node receives at least one message production request of at least one message source node transmission.
When message source node needs message being transferred to message destination node, first message to be transmitted can be added Into message production request, message production request then is sent to first node.During the present invention is implemented, message-oriented middleware cluster In message production process can refer to more copy storing process of message, also i.e. by message storage to message-oriented middleware cluster In each node.Message production request can wrap containing the instruction (command) for executing write operation, can serve to indicate that first Node stores message entrained in message production request.
Of course, it is possible to there is more than one message source node to send message production request, a message source section to first node Point can also send the production request of more than one message.On this basis, first node can receive multiple message productions and ask It asks.
302, first node is based on the corresponding write-in journal entries of each message production request generation and caches, when generation When write-in journal entries reach the first preset number, the write-in journal entries of the first preset number are subjected to persistence processing, are obtained Journal entries are written in target to the first preset number.
Message production request can be added to preset by first node whenever receiving a message production request In the frame for instructing log, a write-in journal entries are generated, and the write-in journal entries can be cached.Only to writing When entering journal entries and being cached, the write-in journal entries that first node caches after restarting will be deleted and possibly can not give for change, And treated that write-in journal entries will not be deleted after first node is restarted for persistence, guarantees the write-in day of persistence Will entry is not lost, and the reliability of message storage is improved.Persistence processing may include write-in disk or store in the database The specific processing of persistence is not construed as limiting Deng, the embodiment of the present invention.
In general, one write-in journal entries of the every generation of first node, it is lasting to carry out to the write-in journal entries Change processing.But persistence processing is carried out to write-in journal entries, the time for needing to consume is longer, in the biggish feelings of request amount Under condition, persistence processing is carried out if one write-in journal entries of every generation, the QPS of message-oriented middleware cluster can compare It is low.Therefore, the embodiment of the invention provides a kind of method for carrying out persistence processing to write-in journal entries, batch carries out lasting Change.That is, when the write-in journal entries of generation reach the first preset number, just by the write-in log item of first preset number Mesh carries out persistence processing, for example, just writing this 100 write-in journal entries when storing 100 write-in journal entries in caching Enter disk.After the completion of persistence processing, corresponding write-in journal entries in caching can be deleted.
It is found by practice, carry out the time-consuming of primary batch persistence and one is written the time-consuming of journal entries persistence It compares, does not grow proportionately, and time-consuming gap is little.The method that batch provided in an embodiment of the present invention carries out persistence subtracts Lack persistence number of processing, can effectively improve the QPS of message-oriented middleware cluster, and then improve message-oriented middleware cluster Treatment effeciency.
In a kind of possible embodiment, for different request amounts, the first preset number of above-mentioned batch persistence It can be different.The corresponding relationship of request amount Yu the first preset number can be previously provided in first node.First node can be with Real-time detection current request amount, or current request amount can be periodically detected.Then, first node can be in request amount and the In the corresponding relationship of one preset number, corresponding first preset number of current request amount is determined, and then be based on first present count Mesh executes the processing of above-mentioned steps 302, and details are not described herein again.The number of request amount is not fixed, and first node is pre-set right It should be related to the corresponding relationship that can also be request amount range Yu the first preset number, be not construed as limiting herein.
For example, corresponding first preset number can also be larger when request amount is larger, guarantee the different request amounts the case where Under, persistence number of processing is roughly the same, maintains higher QPS on the basis of improving the QPS of message-oriented middleware cluster.
303, first node obtains the target of the first preset number from the target of the first preset number write-in journal entries Message production request, the first message that the target message production request of the first preset number of storage carries.
Step 302 and step 303 can be asynchronous implement namely step 302 and step 303 can be real by different processes Existing, step 302 can carry out batch persistence processing next time, without waiting for walking when completing primary batch persistence Rapid 303 release process.In step 303, the write-in log persistently handled has been completed in the available step 302 of first node Entry executes the instruction of the write operation in corresponding message production request, corresponding first message to be transmitted is deposited Storage.Storage processing is also possible to persistence processing, the first message to be transmitted so as to persistent storage.
304, when the write-in journal entries of generation reach the first preset number, first node is at least one second node Transmission is synchronously written request.
In above-mentioned steps 302, when the write-in journal entries of caching reach the first preset number, first node can be closed And journal entries are written in first preset number, generate one and are synchronously written request, then can to its corresponding all the Two nodes send this and are synchronously written request, so that second node stores message.This, which is synchronously written request, can carry above-mentioned The write-in journal entries of one preset number.First node can be synchronously written simultaneously to its corresponding all second node transmission and ask It asks, is also possible to each second node broadcast synchronization write request, to first node to second node in the embodiment of the present invention The concrete mode that transmission is synchronously written request is not construed as limiting.
In a kind of possible embodiment, first node can be requested synchronization removal based at least one second node Processing progress, determine the number of at least one second node untreated synchronization removal request.When untreated synchronization removal When the number of request reaches the second number upper limit, first node, which suspends at least one second node, sends synchronization removal request, Until the number of untreated synchronization removal request is less than the second number upper limit.
First node can recorde be sent to each second node be synchronously written request, when receiving any second node Confirmation message when, by it is corresponding be synchronously written request marks be have confirmed that (committed), in turn, first node can unite Count the untreated number for being synchronously written request.
When the untreated number for being synchronously written request of any second node reaches the first number upper limit (such as 400) When, first node, which can suspend to second node transmission, is synchronously written request, until the untreated number for being synchronously written request Mesh just continues transmission and is synchronously written request less than the first number upper limit.By above-mentioned processing, first node is sent to second node After one is synchronously written request, second node can be not to wait for and send the confirmation message for being synchronously written request, that is, under can be transmitted One is synchronously written request, improves the treatment effeciency of message-oriented middleware cluster.For example, first node can pass through pipeline (pipeline) sends multiple RPC (Remote Procedure Call, remote procedure call) requests, and can count untreated RPC request number.
After first node is synchronously written request to the transmission of at least one second node, corresponding second node can be received together Walk write request.
305, when second node, which receives, is synchronously written request, the write-in journal entries of the first preset number are held Longization processing obtains the target write-in journal entries of the first preset number.
Any second node receives when being synchronously written request, and available this is synchronously written the first present count in request Journal entries are written in purpose, and the write-in journal entries of first preset number are carried out the processing of batch persistence, detailed process With first node step 302 persistence treatment process similarly, details are not described herein again.
306, second node is synchronously written the corresponding write-in confirmation message of request to first node transmission.
When second node completion is lasting to the batch of the write-in journal entries for the first preset number being synchronously written in request When changing processing, write-in confirmation message can be sent to first node.This process withouts waiting for second node completely to be transmitted Message store and complete, it is only necessary to the write-in journal entries batch persistence of the first preset number is handled and is completed, is reduced The interaction of first node and second node is time-consuming, improves the QPS of message-oriented middleware cluster, and then improve message-oriented middleware cluster Treatment effeciency.
307, second node obtains the target of the first preset number from the target of the first preset number write-in journal entries Message production request, the first message that the target message production request of the first preset number of storage carries.
When second node completes the persistence processing to the first preset number in step 305, second node can be obtained Persistence is taken treated target write-in log, executes the instruction of the write operation in corresponding message production request, will correspond to First message to be transmitted stored.Detailed process and the message storing process of first node in step 303 similarly, this Place repeats no more.Since persistence treated target write-in log will not lose since second node is restarted, when the When restarting after two nodes break downs, message can be continued based on the target write-in log for having completed persistence processing and deposited Storage, guarantees that the message having been acknowledged is not lost.
308, first node sends message to the message producer when the write-in confirmation message received meets preset rules Produce success message.
Request is produced for the message that any message source node is sent, by the processing of step 301-307, first node can To receive the write-in confirmation message (confirm) of at least one second node transmission, and then the write-in received can be counted The number of confirmation message.When the write-in confirmation message received meets preset rules, first node can be to the message source section Point sends message and produces success message, which can serve to indicate that the number of write-in confirmation message is more than preset threshold. Preset rules can be write-in confirmation message number be more than message-oriented middleware cluster node total number half, alternatively, may be used also It is more than 2/3rds of the node total number of message-oriented middleware cluster with the number for being write-in confirmation message.The embodiment of the present invention is to tool The preset rules of body are not construed as limiting.For example, preset rules are section of the number more than message-oriented middleware cluster that confirmation message is written When the half of point sum, if the node total number of message-oriented middleware cluster is 3, when first node receives 2 write-in confirmations When thering are 2 node processings to complete in message, namely 3 nodes of composition message-oriented middleware cluster, it can be sent out to message source node Message is sent to produce success message, needing not wait for 3 nodes, all processing is completed.
It can be seen that first node, which withouts waiting for each second node, sends write-in confirmation message, as long as being more than pre- If the second node of threshold value has sent write-in confirmation message, disappear for causing second node to fail to timely feedback write-in confirmation The factor of breath, such as network delay have certain tolerance, improve the reliability of message-oriented middleware cluster.
In the above process after the completion of message production, message destination node can pull message from first node, pull message The detailed process embodiment of the present invention do not introduce.After message destination node is disposed to message, it can enable among message Part cluster deletes the message.The process flow of the method flow diagram that message as shown in connection with fig. 4 is deleted, the method that message is deleted can To include the following steps:
401, first node receives at least one message removal request of at least one message destination node transmission.
After message destination node is disposed to second message, it can send to first node and disappear with to be deleted second The corresponding message removal request of manner of breathing, which can carry the mark of second message to be deleted, for example, disappearing Breath removal request can be an ACK (acknowledgement, confirmation) message, is used to indicate and has handled to second message Finish.
Of course, it is possible to there is more than one message destination node to send message removal request, a message mesh to first node Node can also send more than one message removal request.On this basis, first node can receive multiple message and delete Except request.
402, first node is based on the corresponding deletion journal entries of each message removal request generation and caches, when generation When deletion journal entries reach third preset number, the deletion journal entries of third preset number are subjected to persistence processing, are obtained Target to third preset number deletes journal entries.
Similarly, details are not described herein again for the processing of step 402 and above-mentioned steps 302.Delete the third present count of journal entries Mesh and the first preset number of write-in journal entries may be the same or different, and technical staff, which can according to need, to be set It sets.
403, first node is deleted in journal entries from the target of third preset number, obtains the target of third preset number Message removal request deletes the second message of the target message removal request of third preset number.
Similarly, difference is for the processing of step 403 and above-mentioned steps 303, and can be for carrying in deletion journal entries is deleted Except the instruction of operation, first node can second message progress of the instruction based on the delete operation to persistence processing is completed It deletes.
404, when the deletion journal entries of generation reach third preset number, first node is at least one second node Send synchronization removal request.
Similarly, details are not described herein again for the processing of step 404 and above-mentioned steps 304.
In a kind of possible embodiment, first node can be requested synchronization removal based at least one second node Processing progress, determine the number of at least one second node untreated synchronization removal request.When untreated synchronization removal When the number of request reaches the second number upper limit, suspends at least one second node and send synchronization removal request, until not locating The number of the synchronization removal request of reason is less than the second number upper limit.
After first node sends synchronization removal request at least one second node, corresponding second node can be received together Walk removal request.
405, when second node receives synchronization removal request, the deletion journal entries of third preset number are held Longization processing, the target for obtaining third preset number delete journal entries.
406, second node sends synchronization removal to first node and requests corresponding deletion confirmation message.
407, second node is deleted in journal entries from the target of third preset number, obtains the target of third preset number Message removal request deletes the second message of the target message removal request of third preset number.
Similarly, details are not described herein again for the processing of step 405-407 and above-mentioned steps 305-307.
Any second node is sent after synchronization removal requests corresponding deletions confirmation message to first node, first node with Any second node can delete corresponding deletion journal entries, alternatively, can be by corresponding deletion journal entries It stores in other storage locations, to recall to the deletion journal entries, deletion of the embodiment of the present invention to having confirmed that The specific processing of journal entries is not construed as limiting.
If first node, which needs to send to message destination node, deletes confirmation message, can also be based on and step 308 Method similarly is handled, and details are not described herein again.
Destination node is consumed after pulling message to first node, when completing to the Message Processing, by active to first Node sends the mode of message removal request, explicitly informs that the first node message has handled success.In this way, among message Part cluster can just delete the message by above-mentioned steps 401-407, if the message of consumption destination node is lost, Message can be pulled to first node again by then consuming destination node, it is ensured that before consumption destination node processing is completed, be disappeared Breath will not be lost, and the reliability of message-oriented middleware cluster is improved.
Illustratively, the message-oriented middleware cluster introduced in the above process can be CMQ (Cloud Message Queue, Cloud message queue), CMQ is a kind of highly reliable high performance financial level message-oriented middleware cluster, passes through raft between the node of CMQ (raft) algorithm guarantees highly reliable strongly consistent, it is ensured that the reliability of message storage.Message-oriented middleware cluster signal as shown in Figure 2 Figure, in a message-oriented middleware cluster can by first node (leader, leader node) and second node (follower, with With node) it constitutes, consistency can be set on first node and second node and guaranteed module, instruction log (command ) and state machine (state machine) log.First node can be used as the leader node of at least one second node, and instruction is extremely A few second node carries out storage or delete operation.Message source node, can be to message-oriented middleware collection as the message producer The first node of group sends message production request.Then, first node can be generated message production and request corresponding write-in (entry) journal entries, and guarantee that journal entries storage will be written into instruction log in module by consistency, also, may be used also To guarantee that module will be written journal entries generation and be synchronously written request by consistency, it is sent to each second node, realizes step It is rapid to carry out above-mentioned steps 301-302,304,.Second node receives when being synchronously written request, can guarantee mould by consistency Journal entries storage will be written into instruction log in block, realize step 305-306.Write-in journal entries can be applied in first segment On the state machine of point and second node, the message in the available write-in journal entries of state machine produces request, by message source section Disk is written in the message that point is sent, and updates the ApplyIndex (application parameter) stored in state machine, realizes step 303,307. Finally, first node can feed back write-in confirmation message to message source node, step 308 is realized.
Message destination node can pull message to first node as message consumer, to carry out Message Processing. The deletion process of CMQ and it is above-mentioned similarly, details are not described herein again.Message-oriented middleware clustering performance contrast schematic diagram as shown in Figure 5, The data volume size of every message of CMQ and RabbitMQ processing is all 1KB, CMQ method based on the embodiment of the present invention It is handled, the QPS of CMQ is 4 times of RabbitMQ or so, and performance is far more than RabbitMQ.
In the embodiment of the present invention, first node and second node in message-oriented middleware cluster can be to write-in log items Mesh deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message, subtracts Few number of processing, can reduce the time-consuming of message-oriented middleware cluster, improve the treatment effeciency of message-oriented middleware.Also, first Node can send message to the message producer and produce successfully when the write-in confirmation message satisfaction received meets preset rules Message withouts waiting for all second nodes and all sends write-in confirmation message, in the same of the time-consuming for reducing message-oriented middleware cluster When, the factor for causing second node to fail to timely feedback write-in confirmation message can be improved and disappear with certain tolerance Cease the reliability of middleware cluster.
Based on the same technical idea, the embodiment of the invention also provides a kind of device of message storage, which can be with It is the first node in above-described embodiment.As shown in fig. 6, the device includes:
Receiving module 610, it is described for receiving at least one message production request of at least one message source node transmission Message production request carries first message to be transmitted;
Log persistence module 620, for generating corresponding write-in journal entries based on the production request of each message and delaying It deposits, when the write-in journal entries of generation reach the first preset number, carries described the to the transmission of at least one second node The write-in journal entries of one preset number are synchronously written request, and the write-in journal entries of first preset number are carried out Persistence processing obtains the target write-in journal entries of first preset number;
Memory module 630, for it is pre- to obtain described first from the target of first preset number write-in journal entries If the target message of number produces request, the target message production for storing first preset number requests carry first to disappear Breath.
Optionally, described device further include:
Determining module 640, for, to the processing progress for being synchronously written request, being determined based at least one described second node The untreated number for being synchronously written request of at least one second node;
The log persistence module 620 is also used to reach first when the untreated number for being synchronously written request When the number upper limit, suspends to the transmission of at least one described second node and be synchronously written request, until the untreated synchronous write The number for entering request is less than the first number upper limit.
Optionally, described device further include:
Sending module 650 is used for for any message source node, when the write-in confirmation message received meets preset rules When, Xiang Suoshu message source node sends message and produces success message, wherein said write confirmation message is used to indicate second node Persistence processing is carried out to the message source node corresponding target write-in journal entries to complete.
Optionally, the receiving module 610 is also used to receive at least one that at least one message destination node is sent and disappears Removal request is ceased, the message removal request is corresponding with second message to be deleted;
The log persistence module 620 is also used to generate corresponding deletion log item based on each message removal request Mesh simultaneously caches, and when the deletion journal entries of generation reach third preset number, takes to the transmission of at least one described second node The synchronization removals of deletion journal entries with the third preset number is requested, and by the deletion day of the third preset number Will entry carries out persistence processing, and the target for obtaining the third preset number deletes journal entries;
Described device further include:
It is pre- to obtain the third for deleting in journal entries from the target of the third preset number for removing module 660 If the target message removal request of number deletes the second message of the target message removal request of the third preset number.
Optionally, the determining module 640 is also used to request synchronization removal based at least one described second node Processing progress determines the number of the untreated synchronization removal request of at least one described second node;
The log persistence module 620 is also used to reach second when the number of the untreated synchronization removal request When the number upper limit, suspends at least one described second node and send synchronization removal request, until untreated synchronize is deleted Except the number of request is less than the second number upper limit.
Based on the same technical idea, the embodiment of the invention also provides a kind of device of message storage, which can be with It is the second node in above-described embodiment.As shown in fig. 7, the device includes:
Receiving module 710, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request is carried There are the write-in journal entries of the first preset number, said write journal entries produce request by corresponding message and generate, described to disappear First message to be transmitted is carried in breath production request;
Log persistence module 720, for the write-in journal entries of first preset number to be carried out persistence processing, Obtain the target write-in journal entries of first preset number;
Memory module 730, for it is pre- to obtain described first from the target of first preset number write-in journal entries If the target message of number produces request, the target message production for storing first preset number requests carry first to disappear Breath.
Optionally, described device further include:
Sending module 740, for being synchronously written the corresponding write-in confirmation message of request described in sending to the first node.
Optionally, the receiving module 710 is also used to receive the synchronization removal request that the first node is sent, described The deletion journal entries of third preset number are carried in synchronization removal request;
The log persistence module 720, it is lasting to be also used to carry out the deletion journal entries of the third preset number Change processing, the target for obtaining the third preset number delete journal entries;
Described device further include:
It is pre- to obtain the third for deleting in journal entries from the target of the third preset number for removing module 750 If the target message removal request of number deletes the second message of the target message removal request of the third preset number.
Optionally, the sending module 740, is also used to:
The synchronization removal, which is sent, to the first node requests corresponding deletion confirmation message.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method Embodiment in be described in detail, no detailed explanation will be given here.
In foregoing invention embodiment, first node and second node in message-oriented middleware cluster can be to write-in logs Entry deletes log progress batch persistence processing, corresponding to carry out batch processing when storing or deleting message, Number of processing is reduced, the time-consuming of message-oriented middleware cluster can be reduced, improve the treatment effeciency of message-oriented middleware.
It should be understood that the device of message provided by the above embodiment storage is when message stores, only with above-mentioned each function Can module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different functions Module is completed, i.e., the internal structure of first node or second node is divided into different functional modules, to complete above description All or part of function.In addition, the device of message storage provided by the above embodiment and the embodiment of the method for message storage Belong to same design, specific implementation process is detailed in embodiment of the method, and which is not described herein again.
Fig. 8 is a kind of structural schematic diagram of server provided in an embodiment of the present invention, which can be because of configuration or property Energy is different and generates bigger difference, may include one or more processors (central processing Units, CPU) 801 and one or more memory 802, wherein at least one finger is stored in the memory 802 It enables, at least one instruction is loaded by the processor 801 and executed to realize disappearing for above-mentioned first node or second node Cease the method and step of storage.
In the exemplary embodiment, a kind of computer readable storage medium is additionally provided, the memory for example including instruction, The method that above-metioned instruction can be executed by the processor in server to complete above-mentioned message storage.For example, described computer-readable Storage medium can be ROM, random access memory (RAM), CD-ROM, tape, floppy disk and optical data storage devices etc..
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.

Claims (15)

1. a kind of method of message storage, which is characterized in that the method is applied to the first node of message-oriented middleware cluster, institute The method of stating includes:
Receive at least one message production request of at least one message source node transmission, message production request carry to The first message of transmission;
The corresponding write-in journal entries of request generation are produced based on each message and are cached, when the write-in journal entries of generation reach When the first preset number, the same of the write-in journal entries for carrying first preset number is sent at least one second node Write request is walked, and the write-in journal entries of first preset number are subjected to persistence processing, it is default to obtain described first Journal entries are written in the target of number;
From the target of first preset number write-in journal entries, the target message production of first preset number is obtained Request stores the first message that the target message production request of first preset number carries.
2. the method according to claim 1, wherein the method also includes:
Based at least one described second node to the processing progress for being synchronously written request, at least one described second node is determined The untreated number for being synchronously written request;
When the untreated number for being synchronously written request reaches the first number upper limit, suspend to it is described at least one second Node transmission is synchronously written request, until the untreated number for being synchronously written request is less than the first number upper limit.
3. the method according to claim 1, wherein the method also includes:
For any message source node, when the write-in confirmation message received meets preset rules, Xiang Suoshu message source node It sends message and produces success message, wherein said write confirmation message is used to indicate second node to the message source node pair The target write-in journal entries answered carry out persistence processing and complete.
4. the method according to claim 1, wherein the method also includes:
Receive at least one message destination node transmission at least one message removal request, the message removal request with wait delete The second message removed is corresponding;
Corresponding deletion journal entries are generated based on each message removal request and are cached, when the deletion journal entries of generation reach When third preset number, the deletion journal entries for carrying the third preset number are sent at least one described second node Synchronization removal request, and by the deletion journal entries of the third preset number carry out persistence processing, obtain the third The target of preset number deletes journal entries;
It is deleted in journal entries from the target of the third preset number, the target message for obtaining the third preset number is deleted Request, deletes the second message of the target message removal request of the third preset number.
5. according to the method described in claim 4, it is characterized in that, the method also includes:
Based on the processing progress that at least one described second node requests synchronization removal, at least one described second node is determined The number of untreated synchronization removal request;
When the number of the untreated synchronization removal request reaches the second number upper limit, suspend to it is described at least one second Node sends synchronization removal request, until the number of the untreated synchronization removal request is less than the second number upper limit.
6. a kind of method of message storage, which is characterized in that the method is applied to the second node of message-oriented middleware, the side Method includes:
The request that is synchronously written of first node transmission is received, it is described to be synchronously written the write-in that the first preset number is carried in request Journal entries, said write journal entries produce request by corresponding message and generate, carried in message production request to The first message of transmission;
The write-in journal entries of first preset number are subjected to persistence processing, obtain the target of first preset number Journal entries are written;
From the target of first preset number write-in journal entries, the target message production of first preset number is obtained Request stores the first message that the target message production request of first preset number carries.
7. according to the method described in claim 6, it is characterized in that, the write-in journal entries by first preset number After progress persistence processing, further includes:
Corresponding write-in confirmation message is requested to being synchronously written described in first node transmission.
8. according to the method described in claim 6, it is characterized in that, the method also includes:
The synchronization removal request that the first node is sent is received, carries third preset number in the synchronization removal request Delete journal entries;
The deletion journal entries of the third preset number are subjected to persistence processing, obtain the target of the third preset number Delete journal entries;
It is deleted in journal entries from the target of the third preset number, the target message for obtaining the third preset number is deleted Request, deletes the second message of the target message removal request of the third preset number.
9. according to the method described in claim 8, it is characterized in that, the deletion journal entries by the third preset number After progress persistence processing, further includes:
The synchronization removal, which is sent, to the first node requests corresponding deletion confirmation message.
10. a kind of device of message storage, which is characterized in that described device is the first node of message-oriented middleware cluster, described Device includes:
Receiving module, for receiving at least one message production request of at least one message source node transmission, the message is raw It produces request and carries first message to be transmitted;
Log persistence module works as life for generating corresponding write-in journal entries based on the production request of each message and caching At write-in journal entries reach the first preset number when, at least one second node transmission carry first present count Purpose write-in journal entries are synchronously written request, and the write-in journal entries of first preset number are carried out at persistence Reason obtains the target write-in journal entries of first preset number;
Memory module, for obtaining first preset number from the target of first preset number write-in journal entries Target message produce request, store the first message that the target message production request of first preset number carries.
11. device according to claim 10, which is characterized in that described device further include:
Determining module, for, to the processing progress for being synchronously written request, determination to be described extremely based at least one described second node Few untreated number for being synchronously written request of a second node;
The log persistence module is also used to reach the first number upper limit when the untreated number for being synchronously written request When, suspend to the transmission of described at least one second node and be synchronously written request, untreated is synchronously written request until described Number is less than the first number upper limit.
12. device according to claim 10, which is characterized in that described device further include:
Sending module is used for for any message source node, when the write-in confirmation message received meets preset rules, to institute It states message source node and sends message production success message, wherein said write confirmation message is used to indicate second node to described The corresponding target write-in journal entries of message source node carry out persistence processing and complete.
13. a kind of device of message storage, which is characterized in that described device is the second node of message-oriented middleware, described device Include:
Receiving module, for receiving the request that is synchronously written of first node transmission, described be synchronously written in request carries first The write-in journal entries of preset number, said write journal entries produce request by corresponding message and generate, the message production First message to be transmitted is carried in request;
Log persistence module obtains institute for the write-in journal entries of first preset number to be carried out persistence processing State the target write-in journal entries of the first preset number;
Memory module, for obtaining first preset number from the target of first preset number write-in journal entries Target message produce request, store the first message that the target message production request of first preset number carries.
14. a kind of server, which is characterized in that the server includes processor and memory, is stored in the memory At least one instruction, at least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, institute Code set or instruction set is stated to be loaded by the processor and executed to realize message storage as claimed in claim 1 to 5 Method;Or the method for the message storage as described in claim 6 to 9 is any.
15. a kind of computer readable storage medium, which is characterized in that be stored at least one instruction, extremely in the storage medium A few Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or instruction The method that collection is loaded by the processor and executed to realize message storage as claimed in claim 1 to 5;As right is wanted The method for asking 6 to 9 any message storages.
CN201810791254.8A 2018-07-18 2018-07-18 Message storage method, device and storage medium Active CN110213317B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810791254.8A CN110213317B (en) 2018-07-18 2018-07-18 Message storage method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810791254.8A CN110213317B (en) 2018-07-18 2018-07-18 Message storage method, device and storage medium

Publications (2)

Publication Number Publication Date
CN110213317A true CN110213317A (en) 2019-09-06
CN110213317B CN110213317B (en) 2021-10-29

Family

ID=67779788

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810791254.8A Active CN110213317B (en) 2018-07-18 2018-07-18 Message storage method, device and storage medium

Country Status (1)

Country Link
CN (1) CN110213317B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111880949A (en) * 2020-07-28 2020-11-03 北京奇艺世纪科技有限公司 Method, system, device, server and terminal for processing service information
CN112732493A (en) * 2021-03-30 2021-04-30 恒生电子股份有限公司 Method and device for newly adding node, node of distributed system and storage medium
CN112822227A (en) * 2019-11-15 2021-05-18 北京金山云网络技术有限公司 Data synchronization method, device, equipment and storage medium of distributed storage system
CN113839923A (en) * 2021-08-28 2021-12-24 西安交通大学 Multi-node-oriented high-performance processing method

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7085803B1 (en) * 1996-11-05 2006-08-01 Oracle International Corporation Platform-independent programmable batch processing engine
CN101707633A (en) * 2009-11-27 2010-05-12 山东中创软件商用中间件股份有限公司 Message-oriented middleware persistent message storing method based on file system
CN103488690A (en) * 2013-09-02 2014-01-01 用友软件股份有限公司 Data integrating system and data integrating method
CN103607476A (en) * 2013-12-05 2014-02-26 上海普坤信息科技有限公司 System and method for distributed multi-stage persistence of huge number of messages
CN104731912A (en) * 2015-03-24 2015-06-24 浪潮集团有限公司 Message transmission method and device for message middleware MQ
CN105262831A (en) * 2015-10-30 2016-01-20 北京奇艺世纪科技有限公司 Method, device, and synchronizing system for synchronizing data among memory systems
CN105912628A (en) * 2016-04-07 2016-08-31 北京奇虎科技有限公司 Synchronization method and device for master database and slave database
CN105978968A (en) * 2016-05-11 2016-09-28 山东合天智汇信息技术有限公司 Real-time transmission processing method, server and system of mass data
US20160335287A1 (en) * 2015-05-14 2016-11-17 Alibaba Group Holding Limited Stream computing system and method
CN106598762A (en) * 2016-12-29 2017-04-26 上海理想信息产业(集团)有限公司 Message synchronization method and system

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7085803B1 (en) * 1996-11-05 2006-08-01 Oracle International Corporation Platform-independent programmable batch processing engine
CN101707633A (en) * 2009-11-27 2010-05-12 山东中创软件商用中间件股份有限公司 Message-oriented middleware persistent message storing method based on file system
CN103488690A (en) * 2013-09-02 2014-01-01 用友软件股份有限公司 Data integrating system and data integrating method
CN103607476A (en) * 2013-12-05 2014-02-26 上海普坤信息科技有限公司 System and method for distributed multi-stage persistence of huge number of messages
CN104731912A (en) * 2015-03-24 2015-06-24 浪潮集团有限公司 Message transmission method and device for message middleware MQ
US20160335287A1 (en) * 2015-05-14 2016-11-17 Alibaba Group Holding Limited Stream computing system and method
CN105262831A (en) * 2015-10-30 2016-01-20 北京奇艺世纪科技有限公司 Method, device, and synchronizing system for synchronizing data among memory systems
CN105912628A (en) * 2016-04-07 2016-08-31 北京奇虎科技有限公司 Synchronization method and device for master database and slave database
CN105978968A (en) * 2016-05-11 2016-09-28 山东合天智汇信息技术有限公司 Real-time transmission processing method, server and system of mass data
CN106598762A (en) * 2016-12-29 2017-04-26 上海理想信息产业(集团)有限公司 Message synchronization method and system

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112822227A (en) * 2019-11-15 2021-05-18 北京金山云网络技术有限公司 Data synchronization method, device, equipment and storage medium of distributed storage system
CN112822227B (en) * 2019-11-15 2022-02-25 北京金山云网络技术有限公司 Data synchronization method, device, equipment and storage medium of distributed storage system
CN111880949A (en) * 2020-07-28 2020-11-03 北京奇艺世纪科技有限公司 Method, system, device, server and terminal for processing service information
CN111880949B (en) * 2020-07-28 2024-03-08 北京奇艺世纪科技有限公司 Service information processing method, system, device, server and terminal
CN112732493A (en) * 2021-03-30 2021-04-30 恒生电子股份有限公司 Method and device for newly adding node, node of distributed system and storage medium
CN112732493B (en) * 2021-03-30 2021-06-18 恒生电子股份有限公司 Method and device for newly adding node, node of distributed system and storage medium
CN113839923A (en) * 2021-08-28 2021-12-24 西安交通大学 Multi-node-oriented high-performance processing method

Also Published As

Publication number Publication date
CN110213317B (en) 2021-10-29

Similar Documents

Publication Publication Date Title
CN110213317A (en) The method, apparatus and storage medium of message storage
CN108595207B (en) Gray scale publishing method, rule engine, system, terminal and storage medium
US6529921B1 (en) Dynamic synchronization of tables
WO2022116323A1 (en) Federated learning scheduling method, device, and system
JP2002517858A (en) Collaborative object architecture
US20110055376A1 (en) Heuristic transaction timeout control
CN112162846B (en) Transaction processing method, device and computer readable storage medium
CN113094430B (en) Data processing method, device, equipment and storage medium
CN109388481A (en) A kind of transmission method of transaction information, device, calculates equipment and medium at system
WO2021238301A1 (en) Application update method, apparatus and device and medium
CN110795235A (en) Method and system for deep learning and cooperation of mobile web
CN112492026B (en) Hybrid self-adaptive copy consistency updating method in dynamic cloud storage environment
CN110737510A (en) Block device management system
CN107465725B (en) Heterogeneous long transaction processing system and method based on client information control system
CN106502842B (en) Data reconstruction method and system
CN112463833A (en) Data set acquisition method, system, device and medium
CN108737485A (en) For the method and system of the operation of Internet of Things resource
CN111597035A (en) Simulation engine time advancing method and system based on multiple threads
CN114500352B (en) Plug-in hot updating system and method for medical internet of things message routing device
CN109728937A (en) A kind of method and device of the updating network state of voice mould group
CN115658745A (en) Data processing method, data processing device, computer equipment and computer readable storage medium
CN114610719A (en) Cross-cluster data processing method and device, electronic equipment and storage medium
CN113900821A (en) Task processing method and device, storage medium and electronic equipment
CN106331897B (en) A kind of list processing method and processing device
CN110851445B (en) Method for safely storing data based on block chain technology

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant