CN103209214A - Not only structured query language (NoSQL)-based method for realizing message-oriented middleware - Google Patents
Not only structured query language (NoSQL)-based method for realizing message-oriented middleware Download PDFInfo
- Publication number
- CN103209214A CN103209214A CN201310116696XA CN201310116696A CN103209214A CN 103209214 A CN103209214 A CN 103209214A CN 201310116696X A CN201310116696X A CN 201310116696XA CN 201310116696 A CN201310116696 A CN 201310116696A CN 103209214 A CN103209214 A CN 103209214A
- Authority
- CN
- China
- Prior art keywords
- message
- nosql
- thread
- journal file
- theme
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a not only structured query language (NoSQL)-based method for realizing message-oriented middleware. The method comprises the following steps of: forwarding a cache and restoring shutdown. In a NoSQL, the access control of data is performed in an internal memory, and the read-write speed is high; messages are stored in the NoSQL, so that the high efficiency of message caching can be ensured; when searched, the messages can be positioned quickly by utilizing the advantage of the read-write speed of the internal memory; and by matching with a journal file, the lost of the messages can be avoided after the shutdown of a message-oriented middleware server, and the messages before the shutdown can be restored after restarting.
Description
Technical field
The present invention relates to field of computer technology, relate in particular to a kind of implementation method of the message-oriented middleware based on NoSQL.
Background technology
The message-oriented middleware server, as a high efficiency of transmission and reliable terminal, the transmission of message and storage are the key links.
Message server mainly contains following two kinds of requirements:
1) storage reliably
The applied environment that is suitable for the reliable transmission of message needs.General using file or database are as the storage mode of message, and memory cost is little, and the machine even server is delayed still can recover last message of having stored from lasting storage after restarting.But as storage medium, under the situation of high capacity, frequent I/O operation can reduce the throughput of message-oriented middleware with file or database.
2) transmission efficiently
Suitable very harsh to performance requirement, but message need not reliable lasting applied environment.Carry out efficient buffer memory and the forwarding of message based on internal memory, but after server is delayed machine, can not preserve message.
In addition, message-oriented middleware is very important for data integration and system combination, is each subsystem communication of interface with the middleware, and the complexity that can avoid each species diversity between subsystem to bring is caused as hardware platform, procotol, development language.The high efficiency lasting, transmission of messages of the timeliness of message, message more merits attention.
Propose a kind of NoSQL based on memory in conjunction with journal file at above deficiency, both can solve the performance issue of messenger service, the reliable memory that also gives information.
Summary of the invention
The objective of the invention is in order to overcome the defective of prior art, a kind of implementation method of the message-oriented middleware based on NoSQL is provided, in NoSQL, the access control of data is all carried out in internal memory, and read or write speed is very high.In NoSQL, can guarantee the efficient of message buffering to message stores, when message is searched, can utilize the advantage of memory read-write speed, quickly localization message.Cooperate journal file, can guarantee that more message is not lost after the message-oriented middleware server is delayed machine, and after restarting, can recover the message before the machine of delaying.
A kind of implementation method of the message-oriented middleware based on NoSQL comprises that buffer memory is transmitted and the machine of delaying recovers:
1 buffer memory is transmitted:
The first step: obtain the parameter of message, comprised content and routing mechanism and the service quality etc. of message in the parameter.Mainly contain the content of message, sender and recipient's address or theme, the beginning transmission time, the term of validity, priority has been determined on the memory contents of message-oriented middleware and transmission opportunity by above content.
Second step: parameter is encapsulated and serializing, and the content after the serializing can be stored among journal file and the NoSQL easily, also can draw original various parameters by unserializing from journal file and NoSQL.
The 3rd step: message sends and storage, is handled by four threads (pond).
Thread pool A: monitor the port of appointment, receive message.Qualified message sends immediately, and the message that does not meet the transmission condition is done the persistence processing through issuing thread B after screening.
Thread B: message and mark domain are write journal file, and that mark domain has is newly-increased, revise, deletion.
Thread C: message is write NoSQL.
Thread D: to delayed delivery or send failure but do not exceed at the latest that the message of delivery time resends.
2 machines of delaying recover:
The machine if service routine is delayed, the message before can from journal file, recovering after restarting to restart.Main work for the treatment of is:
1) from the journal file reading of content, mark domain is judged the message before recovering to restart.
2) message after recovering is write file, cover original journal file.
3) message after recovering is deposited in NoSQL.
The beneficial effect that brings based on the message-oriented middleware of NoSQL provided by the invention:
1) utilize NoSQL as buffer memory, fast searching message, the throughput of raising message-oriented middleware.
2) utilize journal file as lasting storage, even when delaying machine, also can guarantee the message of after restarting, recovering all message recovery to be become rational state.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, to do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below, apparently, accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart that system's storage sends message in the embodiment of the invention;
The flow chart that Fig. 2 recovers for the machine of delaying in the invention process.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills belong to the scope of protection of the invention not making the every other embodiment that obtains under the creative work prerequisite.
A kind of implementation method of the message-oriented middleware based on NoSQL comprises that buffer memory is transmitted and the machine of delaying recovers.
Buffer memory is transmitted as shown in Figure 1, is specially:
The first step: the parameter of obtaining message.
Message is made up of two parts, comprises message header and message body.
1) have in the message header:
A) theme of the sender of message and recipient's address or message, can have a plurality of addresses to or theme, each address to or theme to corresponding one group of quality requirement should be arranged.
B) quality of service requirement of message.Comprise beginning transmission time, the term of validity, priority, every group of corresponding address team of quality requirement or theme.
2) message body comprises the message content that needs transmission and storage.A message body can have a plurality of different message headers.
Second step: parameter is encapsulated and serializing.
Available JSON, XML, protobuf carry out serializing and unserializing.
For making things convenient for the transmission of back, calculate the delivery time at the latest according to beginning transmission time and the term of validity here, with the delivery time is replaced the term of validity at the latest.And obtain unique sequence number as key from server, and all the other contents are worth after as serializing, this key-value pair is issued the thread pool A in the 3rd step.
The 3rd step: message sends and storage, is handled by four threads (pond).
Thread pool A: circular treatment, step is as follows:
1) port of monitoring appointment receives message.
2) the message that receives queuing from high to low according to priority, handle in the following order:
If a) current time is in beginning transmission time and delivery time at the latest, then the routing rule according to message sends to receiving terminal.
B) remove stale messages.After if the current time has exceeded at the latest the delivery time or has sent successfully, then delete this delivery time corresponding address to or theme; If the address to or theme all delete, then delete this message.
C) if do not arrive the beginning transmission time, then do not send.
3) message that does not send is added that newly-increased sign sends to thread B.
4) forward 1 to).
Thread B: message and mark domain are write journal file.Circular treatment, step is as follows:
1) receives message.
2) persistence message.Parse the mark domain in the message, sign and message content are write storage file in the mode that increases.If mark domain is newly-increased then this message is sent to thread C.
3) forward 1 to).
Thread C: message is write NoSQL.Circular treatment, step is as follows:
1) receives message.
2) message is write NoSQL.
3) forward 1 to).
Thread D: to delayed delivery or send failure but do not exceed at the latest that the message of delivery time resends, circular treatment, step is as follows:
1) ceases from the NoSQL cancellation, if the current time less than the beginning transmission time, does not then get.
2) if being in, the current time begins transmission time and delivery time, then transmission at the latest.
3) remove stale messages.After if the current time has exceeded at the latest the delivery time or has sent successfully, then delete this delivery time corresponding address to or theme, if the address to or theme all deletion upgrade the content of this message in NoSQL; If the address to or theme all delete, then delete this message from NoSQL.If message is to eliminate entirely, then send the sequence number of deleted marker territory and message to thread B; If part delete the address to or theme, then send the sequence number of modified logo territory and message and not deleted address to or theme to thread B.
4) forward 1 to).
The machine of delaying recovers as shown in Figure 2, is specially:
The machine if service routine is delayed, the message before can from journal file, recovering after restarting to restart.Step is as follows:
1) thread pool A, thread B, thread D wouldn't start.
2) from journal file begin at last read, whenever read delegation, need to judge mark domain.If mark domain is deletion, all the elements of relevant this message can be ignored; If mark domain is newly-increased, this message must be recovered so; If mark domain is deletion, the message header of so current this message hereof needs buffer memory, makes up with message header after reading the message body of this message again.
3) read file header after, the message of recovering is write temporary file with newly-increased mark domain.Write finish after, replace the journal file of former thread B with this temporary file.
4) start thread C, the message after all are recovered sends to thread C.
5) start thread B, thread D, thread pool A in order.
6) this recovery thread withdraws from.
More than the implementation method of a kind of message-oriented middleware based on NoSQL that the embodiment of the invention is provided be described in detail, used specific case herein principle of the present invention and execution mode are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, the part that all can change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (7)
1. the implementation method based on the message-oriented middleware of NoSQL is characterized in that, comprises that buffer memory is transmitted and the machine of delaying recovers, and wherein, buffer memory is transmitted and comprised and parameter is encapsulated and serializing the parameter of obtaining message, and message sends and storage; The machine of delaying recovers to comprise from the journal file reading of content, and mark domain is judged, the message before recovering to restart writes file to the message after recovering, and covers original journal file, and the message after recovering is deposited in NoSQL.
2. method according to claim 1, it is characterized in that, the parameter of message comprises content, routing mechanism and the service quality of message, specifically comprises sender and recipient's address or the theme of message of message, and service quality comprises beginning transmission time, the term of validity, priority.
3. method according to claim 2 is characterized in that, the theme of the sender of message and recipient or message have a plurality of addresses to or theme, each address to or the corresponding corresponding one group of quality requirement of theme.
4. method according to claim 1 is characterized in that, the content after the serializing is stored among journal file and the NoSQL, also can draw original various parameters by unserializing from journal file and NoSQL.
5. method according to claim 1 is characterized in that, message sends and storage, by thread pool and thread process, wherein,
Thread pool A: monitor the port of appointment, receive message, qualified message sends immediately, and the message that does not meet the transmission condition is done the persistence processing through issuing thread B after screening;
Thread B: message and mark domain are write journal file, and that mark domain has is newly-increased, revise, deletion;
Thread C: message is write NoSQL;
Thread D: to delayed delivery or send failure but do not exceed at the latest that the message of delivery time resends.
6. according to claim 1 or 4 described methods, it is characterized in that, carry out serializing and unserializing with JSON, XML, protobuf.
7. according to claim 1 or 2 or 5 described methods, it is characterized in that, when serializing and unserializing, it is the transmission that makes things convenient for the back, calculate the delivery time at the latest according to beginning transmission time and the term of validity, with the delivery time is replaced the term of validity at the latest, and obtain unique sequence number as key from server, and all the other contents are worth after as serializing, this key-value pair is issued thread pool A.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310116696XA CN103209214A (en) | 2013-04-03 | 2013-04-03 | Not only structured query language (NoSQL)-based method for realizing message-oriented middleware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310116696XA CN103209214A (en) | 2013-04-03 | 2013-04-03 | Not only structured query language (NoSQL)-based method for realizing message-oriented middleware |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103209214A true CN103209214A (en) | 2013-07-17 |
Family
ID=48756298
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310116696XA Pending CN103209214A (en) | 2013-04-03 | 2013-04-03 | Not only structured query language (NoSQL)-based method for realizing message-oriented middleware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103209214A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740083A (en) * | 2016-01-28 | 2016-07-06 | 努比亚技术有限公司 | Information processing method, device and system |
| CN107295106A (en) * | 2017-07-31 | 2017-10-24 | 杭州多麦电子商务股份有限公司 | Message data service cluster |
| CN108848031A (en) * | 2018-06-14 | 2018-11-20 | 深圳金证引擎科技有限公司 | Information transferring method and device |
| CN109446216A (en) * | 2018-09-12 | 2019-03-08 | 珠海凡泰极客科技有限责任公司 | A kind of message system for supporting SQL syntax |
| CN109710421A (en) * | 2018-11-16 | 2019-05-03 | 深圳证券交易所 | Recipient's abnormality eliminating method, server and the storage medium of message-oriented middleware |
| CN110943855A (en) * | 2019-11-19 | 2020-03-31 | 山东超越数控电子股份有限公司 | Method for realizing state recovery after shutdown of server through BMC |
| CN111552575A (en) * | 2019-12-31 | 2020-08-18 | 远景智能国际私人投资有限公司 | Message queue-based message consumption method, device and equipment |
| CN111930531A (en) * | 2020-07-01 | 2020-11-13 | 北京奇艺世纪科技有限公司 | Data processing method, data production method, data consumption method, data production device, data consumption device and data consumption medium |
| CN112015523A (en) * | 2020-08-03 | 2020-12-01 | 北京奇艺世纪科技有限公司 | Event loss prevention method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102012944A (en) * | 2010-12-16 | 2011-04-13 | 四川川大智胜软件股份有限公司 | Distributed NOSQL (not only structured query language) database capable of providing replication property |
| CN102279750A (en) * | 2011-08-30 | 2011-12-14 | 浙江大学 | Iterative code generation method based on domain knowledge sharing |
| US20120226658A1 (en) * | 2011-03-04 | 2012-09-06 | Red Hat, Inc. | Data model versioning for document databases |
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN102982130A (en) * | 2012-11-16 | 2013-03-20 | 深圳市融创天下科技股份有限公司 | Synchronized method for not only structured query language (NOSQL) and relational database management system (RDBMS) database and system thereof |
| CN102982075A (en) * | 2012-10-30 | 2013-03-20 | 北京京东世纪贸易有限公司 | Heterogeneous data source access supporting system and method thereof |
-
2013
- 2013-04-03 CN CN201310116696XA patent/CN103209214A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102012944A (en) * | 2010-12-16 | 2011-04-13 | 四川川大智胜软件股份有限公司 | Distributed NOSQL (not only structured query language) database capable of providing replication property |
| US20120226658A1 (en) * | 2011-03-04 | 2012-09-06 | Red Hat, Inc. | Data model versioning for document databases |
| CN102279750A (en) * | 2011-08-30 | 2011-12-14 | 浙江大学 | Iterative code generation method based on domain knowledge sharing |
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN102982075A (en) * | 2012-10-30 | 2013-03-20 | 北京京东世纪贸易有限公司 | Heterogeneous data source access supporting system and method thereof |
| CN102982130A (en) * | 2012-11-16 | 2013-03-20 | 深圳市融创天下科技股份有限公司 | Synchronized method for not only structured query language (NOSQL) and relational database management system (RDBMS) database and system thereof |
Non-Patent Citations (1)
| Title |
|---|
| 沈铢: "NoSQL数据库技术及其应用研究", 《中国优秀硕士学位论文数据库》, 15 September 2012 (2012-09-15) * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740083A (en) * | 2016-01-28 | 2016-07-06 | 努比亚技术有限公司 | Information processing method, device and system |
| CN107295106B (en) * | 2017-07-31 | 2020-08-14 | 杭州多麦电子商务股份有限公司 | Message data service cluster |
| CN107295106A (en) * | 2017-07-31 | 2017-10-24 | 杭州多麦电子商务股份有限公司 | Message data service cluster |
| CN108848031A (en) * | 2018-06-14 | 2018-11-20 | 深圳金证引擎科技有限公司 | Information transferring method and device |
| CN109446216A (en) * | 2018-09-12 | 2019-03-08 | 珠海凡泰极客科技有限责任公司 | A kind of message system for supporting SQL syntax |
| CN109710421A (en) * | 2018-11-16 | 2019-05-03 | 深圳证券交易所 | Recipient's abnormality eliminating method, server and the storage medium of message-oriented middleware |
| CN109710421B (en) * | 2018-11-16 | 2020-12-08 | 深圳证券交易所 | Receiver exception handling method of message middleware, server and storage medium |
| CN110943855A (en) * | 2019-11-19 | 2020-03-31 | 山东超越数控电子股份有限公司 | Method for realizing state recovery after shutdown of server through BMC |
| CN111552575A (en) * | 2019-12-31 | 2020-08-18 | 远景智能国际私人投资有限公司 | Message queue-based message consumption method, device and equipment |
| CN111552575B (en) * | 2019-12-31 | 2023-09-12 | 远景智能国际私人投资有限公司 | Message consumption method, device and equipment based on message queue |
| CN111930531A (en) * | 2020-07-01 | 2020-11-13 | 北京奇艺世纪科技有限公司 | Data processing method, data production method, data consumption method, data production device, data consumption device and data consumption medium |
| CN111930531B (en) * | 2020-07-01 | 2023-08-18 | 北京奇艺世纪科技有限公司 | Data processing, data production and data consumption methods, devices, equipment and media |
| CN112015523A (en) * | 2020-08-03 | 2020-12-01 | 北京奇艺世纪科技有限公司 | Event loss prevention method and device, electronic equipment and storage medium |
| CN112015523B (en) * | 2020-08-03 | 2023-09-01 | 北京奇艺世纪科技有限公司 | Event loss prevention method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103209214A (en) | Not only structured query language (NoSQL)-based method for realizing message-oriented middleware | |
| US9571300B2 (en) | Reducing encapsulation overhead in overlay-based networks | |
| CN106021315B (en) | Log management method and system for application program | |
| US20150189030A1 (en) | Message processing method, device and system for Internet of Things | |
| US9025608B2 (en) | Route switching device and data cashing method thereof | |
| CN104092717A (en) | Message processing method and system, message destination end device and distributed message system | |
| US9769280B2 (en) | Cooperative decentralized caching | |
| CN103095819A (en) | Data information pushing method and data information pushing system | |
| CN102710763B (en) | The method and system of a kind of distributed caching pond, burst and Failure Transfer | |
| CN103500120A (en) | Distributed cache high-availability processing method and system based on multithreading asynchronous double writing | |
| CN105653198A (en) | Data processing method and device | |
| CN105959346A (en) | Data processing system and data processing method based on server cluster | |
| CN106713470A (en) | Distributed cache updating method and cache updating system | |
| CN102142032A (en) | Method and system for reading and writing data of distributed file system | |
| CN110740145A (en) | Message consumption method, device, storage medium and electronic equipment | |
| CN103856516A (en) | Data storage and reading method and data storage and reading device | |
| CN104410655A (en) | Named mechanism based storage system and method of distributed network | |
| US20120059832A1 (en) | Portable Hash-Based Key Location Mechanism | |
| CN102821045B (en) | Method and device for copying multicast message | |
| CN104866528A (en) | Multi-platform data acquisition method and system | |
| CN102891797A (en) | Publishing and subscription method capable of supporting message duration | |
| CN118264617A (en) | Method, system, equipment and storage medium for transmitting data of overlay network | |
| CN102510407A (en) | Method and system for reading and writing microblog | |
| CN104683288A (en) | Message continuous transmission method and device | |
| CN103761255A (en) | Method and system for optimizing data storage of NoSQL mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20130717 |