WO2017008687A1 - 消息队列系统以及实现消息通信的方法 - Google Patents

消息队列系统以及实现消息通信的方法 Download PDF

Info

Publication number
WO2017008687A1
WO2017008687A1 PCT/CN2016/089308 CN2016089308W WO2017008687A1 WO 2017008687 A1 WO2017008687 A1 WO 2017008687A1 CN 2016089308 W CN2016089308 W CN 2016089308W WO 2017008687 A1 WO2017008687 A1 WO 2017008687A1
Authority
WO
WIPO (PCT)
Prior art keywords
buffer
message
standby
persistent storage
storage area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/089308
Other languages
English (en)
French (fr)
Inventor
张俊
何乐
张�杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017008687A1 publication Critical patent/WO2017008687A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication

Definitions

  • the present application relates to the field of computers, and more particularly to a message queuing system and a method of implementing message communication.
  • message queues are an important way to achieve communication between different processes or between different threads in the same process. That is, message transmission is done in the message queue between different processes or between different threads.
  • the message queuing system consists of a cache area on the server where the process (or thread) resides and a local disk.
  • the message pushed by the message producer is first stored in the buffer area.
  • the message in the buffer area is stored in the local disk.
  • only messages stored on the local disk can be further passed to the message consumer.
  • messages stored in the local disk are deleted whether they are delivered to the message consumer, and are deleted after a certain period of time to release the storage space of the local disk.
  • the so-called "reliability" means that the message pushed by the message producer to the message queue system is not lost, even if the message queue system fails, and the message can be saved until it is delivered to the message consumer. In the system until it is passed to the message consumer.
  • the message queuing system consists of the memory on the server where the process (or thread) is located and the database located outside the server.
  • the message pushed by the message producer is first stored in the memory, and then the message is Passed from memory to the message consumer.
  • the messages in the memory are also stored in the database.
  • the message is passed directly from memory to the message consumer, and the IO operation speed of the memory is faster than the IO operation speed of the local disk, so the real-time performance of the message transmission is compared with the first scheme. better.
  • the storage space of the database is relatively large, messages stored in the database are not deleted periodically, so the reliability of the message is better than the first scheme.
  • the inventor of the present application found that at least the following problems exist in the prior art: although in the second scheme, the IO operation speed of the memory is relatively fast, so that the message is transmitted in the message queue system. The real-time performance is better than the first one. However, since the memory does not support simultaneous read/write operations, the message pushed by the message producer is stored in the memory, and the message cannot be transferred from the memory to the message consumer, which greatly affects The real-time nature of message transmission.
  • the message consumer cannot receive the message normally from the message queue system or can only receive the message at a very slow speed, and the message producer produces the message very fast, the more it will be The more messages you get, the more memory you have, and the more serious the memory overflow.
  • the overflowed message is discarded, so that the overflowed message cannot be delivered to the message consumer.
  • the embodiment of the present application provides a message queuing system and a method for implementing message communication, so as to solve the real-time problem in the prior art that the message is transmitted in the message queuing system due to the memory not supporting concurrently performing the read/write operation.
  • sexual problem In addition, the problem that the overflow message caused by the memory overflow in the message queue system in the prior art cannot be normally transmitted to the message consumer is further solved.
  • a message queuing system includes: a first buffer, a second buffer, and a persistent storage area; wherein the first buffer and the second buffer are located in a memory of a server where a message producer and a message consumer are located
  • the second buffer includes a main buffer and a backup buffer;
  • the first buffer is configured to cache a message pushed by a message producer, store a message pushed by the message producer into the persistent storage area, and send a message that has been stored to the persistent storage area to the Said second buffer;
  • the persistent storage area is configured to persistently store the message sent by the first buffer by using a persistent storage medium
  • the backup buffer is configured to buffer a message sent by the first buffer.
  • the main buffer is configured to deliver a message located in the main buffer to a message consumer, where After all the messages of the main buffer are delivered to the message consumer, the active and standby switches are performed with the backup buffer.
  • the backup buffer is further configured to check whether there is a memory overflow before buffering the message sent by the first buffer, and if there is a memory overflow, discard the message sent by the first buffer, and The current status of itself is marked as an abnormal state, and if there is no memory overflow, the message sent by the first buffer is continuously buffered;
  • the main buffer is further configured to check whether the current state of the backup buffer is in a normal state before performing the active/standby switchover with the standby buffer, and if not in the normal state, not in the standby buffer.
  • the active/standby switchover is performed, and the message is read from the persistent storage area, and the read message is delivered to the message consumer. If it is in a normal state, the active/standby switchover is continued with the standby buffer.
  • the main buffer is further configured to: after reading a message from the persistent storage area, determine whether there is a next read message in the backup buffer, and if yes, the preparation The current state of the buffer is marked as normal, and if it does not exist, the current state of the standby buffer is maintained.
  • the persistent storage area is located in a distributed file system external to the server.
  • the main buffer and the backup buffer are isomorphic buffers.
  • the message producer and message consumer are two processes running on the server or two threads running in a process on the server.
  • a method for implementing message communication which first establishes a first buffer and a second buffer in a memory of a server where a message producer and a message consumer are located, and the second buffer includes a main buffer and a backup buffer.
  • the methods include:
  • the first buffer buffers the message pushed by the message producer, and stores the message pushed by the message producer into the persistent storage area to store the message persistently in the persistent storage area;
  • the first buffer sends a message that has been stored to the persistent storage area to the second buffer
  • the backup buffer caches the message sent by the first buffer
  • the main buffer passes the message in the main buffer to the message consumer, and performs the active/standby switchover with the standby buffer after all the messages located in the main buffer are delivered to the message consumer.
  • the method further includes:
  • the backup buffer Before buffering the message sent by the first buffer, the backup buffer checks whether there is a memory overflow. If there is a memory overflow, discards the message sent by the first buffer and marks its current state as an abnormal state;
  • the main buffer Before performing the active/standby switchover with the standby buffer, the main buffer checks whether the current state of the standby buffer is normal. If it is in an abnormal state, it does not perform the active/standby switchover with the standby buffer, but reads from the persistent storage area. Take the message, In order to pass the read message to the message consumer;
  • the message sent by the buffer buffer in the first buffer is:
  • the backup buffer caches the message sent by the first buffer
  • the active/standby switch between the primary buffer and the standby buffer is as follows:
  • the primary buffer and the standby buffer are switched between active and standby.
  • the method further includes:
  • the main buffer After reading the message from the persistent storage area, the main buffer determines whether there is a next read message in the standby buffer, and if so, marks the current state of the standby buffer as a normal state if it does not exist. , maintaining the current state of the standby buffer.
  • the persistent storage area is located in a distributed file system external to the server.
  • the main buffer and the backup buffer are isomorphic buffers.
  • the message producer and message consumer are two processes running on the server or two threads running in a process on the server.
  • a buffer for buffering messages is set in the memory of the server where the message producer and the message consumer are located. Under normal circumstances, the message transmission is completed through the memory access, which ensures the real-time transmission of the message in the message queue system. Moreover, through the dual structure design of the primary and secondary buffers, it is possible to transmit the message from the main buffer to the message consumer while storing the message pushed by the message producer in the backup buffer, thereby further improving the message. Real-time transmission in the message queue system. In addition, the message pushed by the message producer is stored in the persistent storage area to ensure the reliability of the message.
  • the lost message in the memory can also be retrieved from the persistent storage area, thereby avoiding the memory in the message queue system.
  • FIG. 1 schematically illustrates an exemplary application scenario in which embodiments of the present invention may be implemented
  • FIG. 2 is a schematic structural diagram of a message queuing system according to an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of another message queuing system according to an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of another message queuing system according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram showing the operation of a main buffer read message according to an embodiment of the present application.
  • FIG. 6 schematically illustrates a flowchart of a method of implementing message communication in accordance with an embodiment of the present application
  • FIG. 7 schematically illustrates a flow chart of another method of implementing message communication in accordance with an embodiment of the present application.
  • FIG. 8 schematically illustrates a flow chart of another method of implementing message communication in accordance with an embodiment of the present application.
  • FIG. 1 schematically illustrates an exemplary application scenario in which embodiments of the present invention may be implemented.
  • a message producer 10 and a message consumer 20 on the server.
  • the message producer 10 communicates with the message consumer 20, the message producer 10 pushes the message 21 to the message queue system 30, the message queue.
  • System 30 is responsible for communicating message 21 to message consumer 20. Additionally, to ensure the reliability of the message 21, the message queue system 30 also needs to store the message 21 pushed by the message producer 10 into a persistent storage medium, such as a local disk on the server.
  • "delivery” can be expressed as “push” and “pull” modes. The “push” mode is initiated by the message queue system 30 to push the message 21 to the message consumer 20, "pull" mode.
  • the message consumer 20 actively pulling the message 21 from the message queue system 30 by polling.
  • the message producer 10 and the message consumer 20 can be two processes on the server or two threads in the same process on the server.
  • FIG. 1 the schematic diagram shown in Figure 1 is merely one example in which embodiments of the present invention may be implemented.
  • the scope of application of embodiments of the invention is not limited by any aspect of the framework. For example, there are multiple message producers and multiple message consumers on the server, rather than just one message producer and one message consumer as shown in FIG.
  • FIG. 2 is a schematic structural diagram of a message queuing system according to an embodiment of the present application.
  • the message queue system 20 includes a first buffer 21, a second buffer 22, and a persistent storage area 23.
  • the first buffer 21 and the second buffer 22 are located at the server where the message producer and the message consumer are located.
  • the second buffer 22 includes a main buffer 221 and a backup Buffer 222.
  • the internal structure and connection relationship will be further described below in conjunction with the working principle of the device.
  • the first buffer 21 is configured to cache the message pushed by the message producer, store the message pushed by the message producer in the persistent storage area 23, and send the message stored in the persistent storage area 23 to the second buffer. twenty two.
  • the persistent storage area 23 informs the first buffer 21 that the message storage has failed, so that the first buffer 21 re-stores the message in the persistent storage area 23 until the storage is successful, if the storage is successful, the persistent storage The area 23 informs the first buffer 21 that the message storage was successful.
  • the persistent storage area 23 is configured to persistently store the message sent by the first buffer 21 by using the persistent storage medium.
  • the persistent storage area 23 is located in a disk inside the server where the message producer and the message consumer are located, as shown in FIG. 2, or in a database external to the server.
  • the persistent storage area 23 is located in a distributed file system external to the server, as shown in FIG. Arranging the persistent storage area 23 in a distributed file system makes it possible to reliably implement the need for persistent storage of mass messages.
  • Arranging the persistent storage area 23 in a distributed file system makes it possible to reliably implement the need for persistent storage of mass messages.
  • the distributed file system by using the LogFile append method to record messages, it is also possible to avoid the delay problem caused by the remote operation of the distributed file system by the first buffer 21.
  • the buffer 222 is configured to buffer the message sent by the first buffer 21.
  • the main buffer 221 is configured to deliver the message located in the main buffer 221 to the message consumer, and after the message located in the main buffer 221 is all delivered to the message consumer, perform the active/standby switch with the backup buffer 222.
  • the spare buffer 222 is dedicated to buffering messages sent by the first buffer, while the main buffer 221 is dedicated to delivering messages that are located to itself to the message consumer.
  • the main buffer 221 exchanges the identity with the standby buffer, and converts the active/standby relationship, that is, the original main buffer becomes the standby buffer, and starts to buffer the message from the first buffer 21.
  • the original standby buffer becomes the main buffer, passing its cached message to the message consumer.
  • the entire message queue system does not need to read the message backed up in the persistent storage area 23, and all the logic is in three buffers (ie, the first buffer 21, the main buffer 221, and the spare buffer). Completed in zone 222).
  • the main buffer 221 and the backup buffer 222 may be isomorphic buffers, that is, the storage mediums of the two buffers and the storage space are the same size.
  • the main buffer 221 The progress of delivering the message does not match the progress of the buffer buffer 222 to cache the message. The former is faster than the latter. Therefore, before the main buffer 221 passes all of its own messages, the buffer 222 is buffered by the first buffer. The area 21 forwards too many messages and a memory overflow occurs. When a memory overflow occurs, the backup buffer 222 actively discards the overflowed message, and the overflowed message cannot be normally delivered to the message consumer. Therefore, in order to solve the problem that the overflowed message cannot be normally delivered to the message consumer due to the memory overflow of the backup buffer 222, in another optional implementation manner of the present application, as shown in FIG.
  • the buffer 222 is further configured to check whether there is a memory overflow before buffering the message sent by the first buffer 21. If there is a memory overflow, discard the message sent by the first buffer 21 and mark its current status as In an abnormal state, if there is no memory overflow, the message sent by the first buffer 21 is continuously buffered.
  • the main buffer 221 is further configured to check whether the current state of the backup buffer 222 is in a normal state before performing the active/standby switchover with the standby buffer 222. Instead, the message is read from the persistent storage area 23, and the read message is delivered to the message consumer. If it is in the normal state, the active/standby switchover with the standby buffer continues.
  • the main buffer 221 does not perform the active/standby switchover with the standby buffer 222, but directly accesses the persistent storage area 23 (eg, the underlying distributed file system), and the persistent storage area is to be persistent.
  • the message saved in 23 is read and then passed to the message consumer.
  • the main buffer 221 can read a corresponding number of messages from the persistent storage area 23 according to the size of its own storage space, and then deliver the read messages one by one to the message consumer.
  • the main buffer 221 is further configured to: after reading the message from the persistent storage area 23, determine whether there is a next read message in the backup buffer 222. If yes, the progress of the main buffer 221 to deliver the message may be And preparation
  • the buffer 222 caches the progress of the message, and the main buffer 221 marks the current state of the backup buffer 222 as a normal state. If not, the current state of the standby buffer 222 is maintained.
  • the backup buffer 222 caches the message with the identifier 30-50, and the message identified as 1-29 is discarded by the backup buffer 222 due to the memory overflow.
  • the main buffer 221 is firstly taken from the bottom layer. The message is read in the distributed file system. It is assumed that the main buffer 221 can only cache 20 messages at a time, and the main buffer 221 reads the messages identified as 1-20 from the distributed file system, and transmits them to the messages one by one. Message consumer. After the reading is completed, the main buffer 221 determines whether the next read message (ie, the message identified as 21) is in the backup buffer 222. After the judgment, it is found that the next read message with the identifier 21 is not present.
  • the next read message ie, the message identified as 21
  • the main buffer 221 continues to maintain the current state flag of the spare buffer 222, that is, an abnormal state.
  • the main buffer 221 passes all of its own messages, it immediately checks the current state of the standby buffer 222. Since the current state of the checked standby buffer 222 is still abnormal, it continues from the underlying distributed file system. The message is read, ie the messages identified as 21-40 are read and passed to the message consumer one by one. After reading, the main buffer 221 continues to determine whether the next read message (ie, the message identified as 41) is in the backup buffer 222. After the judgment, the next read message with the identifier 41 is read. In the spare buffer 222, the main buffer 221 marks the current state of the spare buffer 222 as a normal state. After the main buffer 221 transmits its own message again, the current state of the checked buffer 222 is normal, so the active/standby switch can be normally performed with the backup buffer 222.
  • the message producer and the message consumer are two processes running on the server or two threads running in a process on the server.
  • a buffer for buffering messages is set in the memory of the server where the message producer and the message consumer are located. Under normal circumstances, the message transmission is completed through the memory access, which ensures the real-time transmission of the message in the message queue system. Moreover, through the dual structure design of the primary and secondary buffers, it is possible to transmit the message from the main buffer to the message consumer while storing the message pushed by the message producer in the backup buffer, thereby further improving the message. Real-time transmission in the message queue system. In addition, the message pushed by the message producer is stored in the persistent storage area to ensure the reliability of the message.
  • the lost message in the memory can also be retrieved from the persistent storage area, thereby avoiding the memory in the message queue system.
  • FIG. 6 is a flow chart schematically showing a method for implementing message communication according to an embodiment of the present application.
  • the method may be performed by the message queue system 20, and the method may include the following steps, for example:
  • Step 600 Establish a first buffer and a second buffer in advance in a memory of a server where the message producer and the message consumer are located, and the second buffer includes a main buffer and a backup buffer.
  • Step 601 The first buffer caches the message pushed by the message producer, and stores the message pushed by the message producer into the persistent storage area to store the message persistently in the persistent storage area.
  • Step 602 The first buffer sends a message that has been stored to the persistent storage area to the second buffer.
  • Step 603 The backup buffer caches the message sent by the first buffer.
  • Step 604 The main buffer passes the message located in the main buffer to the message consumer.
  • Step 605 The main buffer performs an active/standby switchover with the backup buffer after all the messages located in the main buffer are delivered to the message consumer.
  • the method may include the following steps, for example:
  • Step 600 Establish a first buffer and a second buffer in advance in a memory of a server where the message producer and the message consumer are located, and the second buffer includes a main buffer and a backup buffer.
  • Step 601 The first buffer caches the message pushed by the message producer, and stores the message pushed by the message producer into the persistent storage area to store the message persistently in the persistent storage area.
  • Step 602 The first buffer sends a message that has been stored to the persistent storage area to the second buffer.
  • Step 603a The standby buffer checks whether there is a memory overflow. If there is a memory overflow, the process proceeds to step 606. If there is no memory overflow, the process proceeds to step 603.
  • Step 603 The backup buffer caches the message sent by the first buffer.
  • Step 604 The main buffer passes the message located in the main buffer to the message consumer.
  • Step 605a After all the messages located in the main buffer are delivered to the message consumer, the main buffer checks whether the current state of the backup buffer is in a normal state. If it is in an abnormal state, the process proceeds to step 607, if it is a normal state. Go to step 605.
  • Step 605 The primary buffer and the standby buffer perform the active/standby switchover, and the process ends.
  • Step 606 The backup buffer discards the message sent by the first buffer, and marks its current state as an abnormal state, and ends the process.
  • Step 607 The main buffer does not perform the active/standby switchover with the standby buffer, and reads the message from the persistent storage area to deliver the read message to the message consumer, and ends the process.
  • the method may include the following steps, for example:
  • Step 600 Establish a first buffer and a second buffer in advance in a memory of a server where the message producer and the message consumer are located, and the second buffer includes a main buffer and a backup buffer.
  • Step 601 The first buffer caches the message pushed by the message producer, and stores the message pushed by the message producer into the persistent storage area to store the message persistently in the persistent storage area.
  • Step 602 The first buffer sends a message that has been stored to the persistent storage area to the second buffer.
  • Step 603a The standby buffer checks whether there is a memory overflow. If there is a memory overflow, the process proceeds to step 606. If there is no memory overflow, the process proceeds to step 603.
  • Step 603 The backup buffer caches the message sent by the first buffer.
  • Step 604 The main buffer passes the message located in the main buffer to the message consumer.
  • Step 605a After all the messages located in the main buffer are delivered to the message consumer, the main buffer checks whether the current state of the backup buffer is in a normal state. If it is in an abnormal state, the process proceeds to step 607, if it is a normal state. Go to step 605.
  • Step 605 The primary buffer and the standby buffer perform the active/standby switchover, and the process ends.
  • Step 606 The backup buffer discards the message sent by the first buffer, and marks its current state as an abnormal state, and ends the process.
  • Step 607 The main buffer does not perform the active/standby switchover with the standby buffer, and reads the message from the persistent storage area to deliver the read message to the message consumer.
  • Step 608 The main buffer determines whether there is a next read message in the spare buffer. If yes, the process proceeds to step 609. If not, the process proceeds to step 610.
  • Step 609 Mark the current state of the backup buffer as a normal state, and end the process.
  • Step 610 Maintain the current state of the backup buffer and end the process.
  • the persistent storage area is located in a distributed file system external to the server.
  • the primary buffer and the secondary buffer are isomorphic buffers.
  • the message producer and the message consumer are two processes running on the server or two threads running in a process on the server.
  • a buffer for buffering messages is set in the memory of the server where the message producer and the message consumer are located. Under normal circumstances, the message transmission is completed through the memory access, which ensures the real-time transmission of the message in the message queue system. Moreover, through the dual structure design of the primary and secondary buffers, it is possible to store the message pushed by the message producer in the standby At the same time as the buffer, the message is transmitted from the main buffer to the message consumer, which further improves the real-time transmission of the message in the message queue system. In addition, the message pushed by the message producer is stored in the persistent storage area to ensure the reliability of the message.
  • the lost message in the memory can also be retrieved from the persistent storage area, thereby avoiding the memory in the message queue system.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined. Or it can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may be or may be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware and can be implemented in the form of a software functional unit.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Hardware Redundancy (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

一种消息队列系统和实现消息通信的方法，所述系统包括：第一缓冲区（21）、第二缓冲区（22）和持久化存储区（23）；第一缓冲区（21）和第二缓冲区（22）位于消息生产者和消息消费者所在的服务器的内存中，第二缓冲区（22）包括主缓冲区（221）和备缓冲区（222）；第一缓冲区（21）用于缓存消息生产者推送的消息并存储在持久化存储区（23）中，将已存储到持久化存储区（23）的消息发送给第二缓冲区（22）；持久化存储区（23）用于对第一缓冲区（21）发送的消息进行持久化存储；备缓冲区（222）用于缓存第一缓冲区（21）发送的消息；主缓冲区（221）用于将自身的消息传递给消息消费者，在将自身的消息全部传递给消息消费者之后，与备缓冲区（222）进行主备切换。所述系统提高了消息在消息队列系统中传输的实时性。

Description

消息队列系统以及实现消息通信的方法
本申请要求2015年07月14日递交的申请号为201510413284.1、发明名称为“消息队列系统以及实现消息通信的方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机领域，特别是涉及消息队列系统以及实现消息通信的方法。
背景技术
在计算机系统中，消息队列是实现不同进程间通信或者同一进程中的不同线程间通信的一种重要方式。也就是说，不同的进程之间或不同的线程之间在消息队列中完成消息传输。
在现有的相关技术中，已经存在多种关于消息队列系统的实现方案。其中一个方案为，消息队列系统由进程(或线程)所在的服务器上的缓存区以及本地磁盘构成。其中，先将消息生产者推送的消息存储在该缓存区中，当该缓存区中的消息个数达到配置值或者消息的发布时间超过阈值时，再将该缓存区中的消息存储到本地磁盘中，以保证消息的可靠性。并且只有存储到本地磁盘中的消息才可以被进一步传递给消息消费者。另外，由于本地磁盘的存储空间有限，存储在本地磁盘中的消息无论是否被传递给消息消费者，其在保留一定时间后就会被删除，以便释放本地磁盘的存储空间。所谓“可靠性”就是指，消息生产者推送到消息队列系统中的消息不丢失，即使消息队列系统故障也可以被恢复；并且，消息在未被传递给消息消费者之前可以一直保存在消息队列系统中，直到被传递给消息消费者为止。
在第一个方案中，只有位于本地磁盘的消息才可以被传递给消息消费者，而消息在被存储到本地磁盘之前，还必须先被存储在缓存区中，因此，消息传输的实时性会比较差。并且，消息是从本地磁盘传递给消息消费者的，受到本地磁盘的IO操作速度的影响，消息传输的实时性就会更差。另外，由于本地磁盘中的消息会被定期删除，因此消息的可靠性也非常差。
另一个方案为：消息队列系统由进程(或线程)所在的服务器上的内存以及位于该服务器外部的数据库构成。其中，先将消息生产者推送的消息存储在内存中，再将消息 从内存中传递给消息消费者。并且，为了保证消息的可靠性，还将内存中的消息存储到数据库中。显然，在第二个方案中，消息直接从内存传递到消息消费者，并且，内存的IO操作速度比本地磁盘的IO操作速度快，因此，相较于第一个方案，消息传输的实时性更好。另外，由于数据库的存储空间比较大，存储在数据库中的消息不会被定期删除，因此，相较于第一个方案，消息的可靠性也更好。
但是，在实现本申请的过程中，本申请的发明人发现现有技术中至少存在如下问题：虽然在第二个方案中，内存的IO操作速度比较快，使其消息在消息队列系统中传输的实时性相较于第一个方案好。但是，由于内存不支持同时执行读/写操作，因此，在将消息生产者推送的消息存储在内存中的同时，就不能将消息从内存中传递给消息消费者，这样会很大程度地影响消息传输的实时性。
另外，在第二个方案中，如果消息消费者无法正常地从消息队列系统接收消息或者只能以很慢的速度接收消息，并且消息生产者生产消息的速度又非常快，这样就会有越来越多的消息占满内存，严重时会发生内存溢出。当发生内存溢出时，溢出的消息会被丢弃掉，以至于溢出的消息无法正常传递给消息消费者。
发明内容
为了解决上述技术问题，本申请实施例提供了消息队列系统、以及实现消息通信的方法，以解决现有技术中由于内存不支持同时执行读/写操作而影响消息在消息队列系统中传输的实时性的问题。另外，进一步解决现有技术中由于消息队列系统中的内存发生内存溢出而导致溢出的消息无法正常地被传递给消息消费者的问题。
本申请实施例公开了如下技术方案：
一种消息队列系统，包括：第一缓冲区、第二缓冲区以及持久化存储区；其中，所述第一缓冲区和第二缓冲区位于消息生产者和消息消费者所在的服务器的内存中，所述第二缓冲区包括主缓冲区和备缓冲区；
所述第一缓冲区，用于缓存消息生产者推送的消息，将消息生产者推送的消息存储到所述持久化存储区中，并将已存储到所述持久化存储区的消息发送给所述第二缓冲区；
所述持久化存储区，用于通过持久化存储介质对所述第一缓冲区发送的消息进行持久化存储；
所述备缓冲区，用于缓存所述第一缓冲区发送的消息；
所述主缓冲区，用于将位于所述主缓冲区的消息传递给消息消费者，在将位于所述 主缓冲区的消息全部传递给消息消费者之后，与所述备缓冲区进行主备切换。
优选的，所述备缓冲区还用于，在缓存所述第一缓冲区发送的消息之前，检查自身是否存在内存溢出，如果存在内存溢出，丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态，如果不存在内存溢出，继续缓存所述第一缓冲区发送的消息；
所述主缓冲区还用于，在与所述备缓冲区进行主备切换之前，检查所述备缓冲区的当前状态是否为正常状态，如果为非正常状态，不与所述备缓冲区进行主备切换，而从所述持久化存储区中读取消息，并将读取的消息传递给消息消费者，如果为正常状态，继续与所述备缓冲区进行主备切换。
优选的，所述主缓冲区还用于，在从所述持久化存储区中读取消息之后，判断在所述备缓冲区中是否存在下一个读取的消息，如果存在，将所述备缓冲区的当前状态标记为正常状态，如果不存在，维持所述备缓冲区的当前状态。
优选的，所述持久化存储区位于所述服务器外部的分布式文件系统中。
优选的，所述主缓冲区和备缓冲区为同构的缓冲区。
优选的，所述消息生产者和消息消费者为运行在所述服务器上的两个进程或者为运行在所述服务器上的一个进程中的两个线程。
一种实现消息通信的方法，预先在消息生产者和消息消费者所在的服务器的内存中建立第一缓冲区和第二缓冲区，所述第二缓冲区包括主缓冲区和备缓冲区，所述方法包括：
第一缓冲区缓存消息生产者推送的消息，将消息生产者推送的消息存储到持久化存储区中，以便将所述消息持久化地存储在所述持久化存储区中；
第一缓冲区将已存储到所述持久化存储区的消息发送给第二缓冲区；
备缓冲区缓存第一缓冲区发送的消息；
主缓冲区将位于主缓冲区的消息传递给消息消费者，在将位于所述主缓冲区的消息全部传递给消息消费者之后，与备缓冲区进行主备切换。
优选的，所述方法还包括：
在缓存第一缓冲区发送的消息之前，备缓冲区检查自身是否存在内存溢出，如果存在内存溢出，丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态；
在与备缓冲区进行主备切换之前，主缓冲区检查备缓冲区的当前状态是否为正常状态，如果为非正常状态，不与备缓冲区进行主备切换，而从持久化存储区中读取消息， 以便将读取的消息传递给消息消费者；
则所述备缓冲区缓存第一缓冲区发送的消息为：
如果不存在内存溢出，备缓冲区缓存第一缓冲区发送的消息；
则所述主缓冲区与备缓冲区进行主备切换为：
如果为正常状态，主缓冲区与备缓冲区进行主备切换。
优选的，所述方法还包括：
在从持久化存储区中读取消息之后，主缓冲区判断在备缓冲区中是否存在下一个读取的消息，如果存在，将所述备缓冲区的当前状态标记为正常状态，如果不存在，维持所述备缓冲区的当前状态。
优选的，所述持久化存储区位于所述服务器外部的分布式文件系统中。
优选的，所述主缓冲区和备缓冲区为同构的缓冲区。
优选的，所述消息生产者和消息消费者为运行在所述服务器上的两个进程或者为运行在所述服务器上的一个进程中的两个线程。
由上述实施例可以看出，与现有技术相比，本申请的优点在于：
在消息生产者和消息消费者所在的服务器的内存中设置用于缓存消息的缓冲区，在正常情况下，通过内存访问完成消息传输，保证了消息在消息队列系统中传输的实时性。并且，通过主、备缓冲区的双结构设计，可以实现在将消息生产者推送的消息存储在备缓冲区的同时，将消息从主缓冲区中传递给消息消费者，更进一步提高了消息在消息队列系统中传输的实时性。另外，还将消息生产者推送的消息存储在持久化存储区中，保证了消息的可靠性。
此外，当消息队列系统中的备缓冲区出现内存溢出问题时，即，在非正常情况下，也可以从持久化存储区中取回内存中丢失的消息，避免了由于消息队列系统中的内存发生内存溢出而导致溢出的消息无法正常地被传递给消息消费者的问题。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1示意性地示出了本发明的实施方式可以在其中实施的示例性应用场景；
图2示意性地示出了根据本申请实施例的一种消息队列系统的结构示意图；
图3示意性地示出了根据本申请实施例的另一种消息队列系统的结构示意图；
图4示意性地示出了根据本申请实施例的另一种消息队列系统的结构示意图；
图5示性意地示出了根据本申请实施例的主缓冲区读取消息的工作示意图；
图6示意性地示出了根据本申请实施例的一种实现消息通信的方法的流程图；
图7示意性地示出了根据本申请实施例的另一种实现消息通信的方法的流程图；
图8示意性地示出了根据本申请实施例的另一种实现消息通信的方法的流程图。
具体实施方式
参阅图1所示，图1示意性地示出了本发明的实施方式可以在其中实施的示例性应用场景。其中，服务器上存在一个消息生产者10和一个消息消费者20，当消息生产者10与消息消费者20之间进行通信时，消息生产者10将消息21推送到消息队列系统30中，消息队列系统30负责将消息21传递给消息消费者20。另外，为了保证消息21的可靠性，消息队列系统30还需要将消息生产者10推送的消息21存储到持久化存储介质中，如，服务器上的本地磁盘。其中，“传递”可以表现为“推(push)”和“拉(pull)”两种模式，“推”模式是由消息队列系统30主动将消息21推送给消息消费者20，“拉”模式是消息消费者20主动通过轮询的方式从消息队列系统30中拉取消息21。消息生产者10和消息消费者20可以为服务器上的两个进程，也可以为服务器上同一个进程中的两个线程。本领域技术人员可以理解，图1所示的示意图仅是本发明的实施方式可以在其中得以实现的一个示例。本发明实施方式的应用范围不受到该框架任何方面的限制。例如，服务器上存在多个消息生产者和多个消息消费者，而不是如图1所示的仅存在一个消息生产者和一个消息消费者。
为使本申请的上述目的、特征和优点能够更加明显易懂，下面结合附图对本申请实施例进行详细描述。
装置实施例
本申请实施例还提供了一种消息队列系统。请参阅图2，图2示意性地示出了根据本申请实施例的一种消息队列系统的结构示意图。其中，该消息队列系统20包括：第一缓冲区21、第二缓冲区22以及持久化存储区23，第一缓冲区21和第二缓冲区22位于消息生产者和消息消费者所在的服务器的内存中，第二缓冲区22包括主缓冲区221和备 缓冲区222。下面结合该装置的工作原理进一步介绍其内部结构以及连接关系。
第一缓冲区21，用于缓存消息生产者推送的消息，将消息生产者推送的消息存储在持久化存储区23中，并将已存储到持久化存储区23的消息发送给第二缓冲区22。
其中，在消息生产者生产的消息到达第一缓冲区21之后，会被首先持久化地存储到持久化存储区23中，以保证该消息的可靠性。如果存储失败，持久化存储区23告知第一缓冲区21消息存储失败，以便第一缓冲区21重新将消息持久化存储到持久化存储区23中直到存储成功为止，如果存储成功，持久化存储区23告知第一缓冲区21消息存储成功。
持久化存储区23，用于通过持久化存储介质对第一缓冲区21发送的消息进行持久化存储。
在本申请的一个可选实施方式中，持久化存储区23位于消息生产者和消息消费者所在的服务器内部的磁盘中，如图2所示，也可以位于该服务器外部的数据库中。
在本申请的另一个可选实施方式中，持久化存储区23位于该服务器外部的分布式文件系统中，如图3所示。将持久化存储区23布置在分布式文件系统中，可以很可靠地实现对海量消息的持久化存储的需求。另外，在分布式文件系统中，通过LogFile追加方式记录消息，也能够避免因为第一缓冲区21远程操作分布式文件系统而带来的延迟问题。
备缓冲区222，用于缓存第一缓冲区21发送的消息。
主缓冲区221，用于将位于主缓冲区221的消息传递给消息消费者，在将位于主缓冲区221的消息全部传递给消息消费者之后，与备缓冲区222进行主备切换。
也就是说，备缓冲区222专用于缓存第一缓冲区发送的消息，而主缓冲区221专用于将位于自身的消息传递给消息消费者。一旦主缓冲区221将自身的消息全部传递出去之后，就与备缓冲区互换身份，转换主备关系，即，原来的主缓冲区成为备缓冲区，开始缓存来自第一缓冲区21的消息，而原来的备缓冲区成为主缓冲区，将其缓存的消息传递给消息消费者。在以上的正常场景下，整个消息队列系统不需要读取在持久化存储区23中备份的消息，所有的逻辑在三个缓冲区(即，第一缓冲区21、主缓冲区221和备缓冲区222)中完成。
可以理解的，当在服务器的内存中设置用于缓存消息的缓冲区时，通过主、备缓冲区的双结构设计，可以实现在将消息生产者推送的消息存储在备缓冲区的同时，将消息从主缓冲区中传递给消息消费者，更好地保证了消息在消息队列系统中传输的实时性。
在本申请的一个可选实施方式中，主缓冲区221和备缓冲区222可以为同构的缓冲区，即，两个缓冲区的存储介质以及存储空间大小都相同。
考虑到消息消费者有时可能无法正常从消息队列系统中接收消息，或者只能以很慢的速度接收消息，并且消息生产者生产消息的速度又非常快，在这种情况下，主缓冲区221传递消息的进度与备缓冲区222缓存消息的进度会出现不匹配，前者快于后者，因此，在主缓冲区221将自身的消息全部传递出去之前，备缓冲区222就会因为第一缓冲区21转发过多的消息而发生内存溢出，当发生内存溢出时，备缓冲区222就会主动丢弃掉溢出的消息，而导致溢出的消息无法正常被传递给消息消费者。因此，为了解决由于备缓冲区222发生内存溢出而导致溢出的消息无法正常地被传递给消息消费者的问题，在本申请的另一个可选实施方式中，如图4所示，
备缓冲区222还用于，在缓存第一缓冲区21发送的消息之前，检查自身是否存在内存溢出，如果存在内存溢出，丢弃第一缓冲区21发送的消息，并将自身的当前状态标记为非正常状态，如果不存在内存溢出，继续缓存第一缓冲区21发送的消息。
主缓冲区221还用于，在与备缓冲区222进行主备切换之前，检查备缓冲区222的当前状态是否为正常状态，如果为非正常状态，不与备缓冲区222进行主备切换，而是从持久化存储区23中读取消息，并将读取的消息传递给消息消费者，如果为正常状态，继续与备缓冲区进行主备切换。
也就是说，当备缓冲区222发生内存溢出时，其被标记为非正常状态，表明当前的备缓冲区222所缓存的消息不完整，由于某些原因备缓冲区222已经丢弃了部分消息。因此，在这种非正常场景下，主缓冲区221不与备缓冲区222进行主备切换，而是直接访问持久化存储区23(如，底层的分布式文件系统)，将持久化存储区23中保存的消息读取出来后继续传递给消息消费者。其中，主缓冲区221可以根据自己的存储空间大小从持久化存储区23中读取相应数量的消息，然后将读取的消息逐个地传递给消息消费者。
这种方式可以一直持续到主缓冲区221发现自己传递消息的进度可以与备缓冲区222缓存消息的进度匹配，此时主缓冲区221则不再需要从持久化存储区23中读取消息，而是回到正常场景下，通过主备切换方式传递消息。因此，在本申请的另一个可选实施方式中：
主缓冲区221还用于，在从持久化存储区23中读取消息之后，判断在备缓冲区222中是否存在下一个读取的消息，如果存在，说明主缓冲区221传递消息的进度可以与备 缓冲区222缓存消息的进度匹配，主缓冲区221将备缓冲区222的当前状态标记为正常状态，如果不存在，维持备缓冲区222的当前状态。
例如，如图5所示，备缓冲区222缓存了标识为30-50的消息，而标识为1-29的消息由于内存溢出而被备缓冲区222丢弃掉了，主缓冲区221先从底层的分布式文件系统中读取消息，假设主缓冲区221一次性只能缓存20个消息，主缓冲区221从分布式文件系统中读取出标识为1-20的消息，并逐个地传递给消息消费者。在读取完之后，主缓冲区221判断其下一个读取的消息(即标识为21的消息)是否在备缓冲区222中，经过判断发现，其下一个读取的标识为21的消息不在备缓冲区222中，主缓冲区221继续维持备缓冲区222的当前状态标记，即，非正常状态。当主缓冲区221将自身的消息全部传递出去后，即刻检查备缓冲区222的当前状态，由于检查出的备缓冲区222的当前状态仍然为非正常状态，因此继续从底层的分布式文件系统中读取消息，即读取标识为21-40的消息，并逐个地传递给消息消费者。在读取完之后，主缓冲区221继续判断其下一个读取的消息(即标识为41的消息)是否在备缓冲区222中，经过判断发现，其下一个读取的标识为41的消息在备缓冲区222中，主缓冲区221将备缓冲区222的当前状态标记为正常状态。当主缓冲区221将自身的消息再一次全部传递出去后，由于检查出的备缓冲区222的当前状态为正常状态，因此可以正常地与备缓冲区222进行主备切换。
在本申请的一个可选实施方式中，消息生产者和消息消费者为运行在服务器上的两个进程或者为运行在服务器上的一个进程中的两个线程。
由上述实施例可以看出，与现有技术相比，本申请的优点在于：
在消息生产者和消息消费者所在的服务器的内存中设置用于缓存消息的缓冲区，在正常情况下，通过内存访问完成消息传输，保证了消息在消息队列系统中传输的实时性。并且，通过主、备缓冲区的双结构设计，可以实现在将消息生产者推送的消息存储在备缓冲区的同时，将消息从主缓冲区中传递给消息消费者，更进一步提高了消息在消息队列系统中传输的实时性。另外，还将消息生产者推送的消息存储在持久化存储区中，保证了消息的可靠性。
此外，当消息队列系统中的备缓冲区出现内存溢出问题时，即，在非正常情况下，也可以从持久化存储区中取回内存中丢失的消息，避免了由于消息队列系统中的内存发生内存溢出而导致溢出的消息无法正常地被传递给消息消费者的问题。
方法实施例
请参阅图6，图6示意性地示出了根据本申请实施例的一种实现消息通信的方法的流程图，例如，该方法可以由消息队列系统20执行，该方法例如可以包括以下步骤：
步骤600：预先在消息生产者和消息消费者所在的服务器的内存中建立第一缓冲区和第二缓冲区，所述第二缓冲区包括主缓冲区和备缓冲区。
步骤601：第一缓冲区缓存消息生产者推送的消息，将消息生产者推送的消息存储到持久化存储区中，以便将所述消息持久化地存储在所述持久化存储区中。
步骤602：第一缓冲区将已存储到所述持久化存储区的消息发送给第二缓冲区。
步骤603：备缓冲区缓存第一缓冲区发送的消息。
步骤604：主缓冲区将位于主缓冲区的消息传递给消息消费者。
步骤605：主缓冲区在将位于所述主缓冲区的消息全部传递给消息消费者之后，与备缓冲区进行主备切换。
如图7所示，在本申请的一个可选实施方式中，该方法例如可以包括以下步骤：
步骤600：预先在消息生产者和消息消费者所在的服务器的内存中建立第一缓冲区和第二缓冲区，所述第二缓冲区包括主缓冲区和备缓冲区。
步骤601：第一缓冲区缓存消息生产者推送的消息，将消息生产者推送的消息存储到持久化存储区中，以便将所述消息持久化地存储在所述持久化存储区中。
步骤602：第一缓冲区将已存储到所述持久化存储区的消息发送给第二缓冲区。
步骤603a：备缓冲区检查自身是否存在内存溢出，如果存在内存溢出，进入步骤606，如果不存在内存溢出，进入步骤603。
步骤603：备缓冲区缓存第一缓冲区发送的消息。
步骤604：主缓冲区将位于主缓冲区的消息传递给消息消费者。
步骤605a：在将位于所述主缓冲区的消息全部传递给消息消费者之后，主缓冲区检查备缓冲区的当前状态是否为正常状态，如果为非正常状态，进入步骤607，如果为正常状态，进入步骤605。
步骤605：主缓冲区与备缓冲区进行主备切换，结束流程。
步骤606：备缓冲区丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态，结束流程。
步骤607：主缓冲区不与备缓冲区进行主备切换，而从持久化存储区中读取消息，以便将读取的消息传递给消息消费者，结束流程。
如图8所示，在本申请的另一个可选实施方式中，该方法例如可以包括以下步骤：
步骤600：预先在消息生产者和消息消费者所在的服务器的内存中建立第一缓冲区和第二缓冲区，所述第二缓冲区包括主缓冲区和备缓冲区。
步骤601：第一缓冲区缓存消息生产者推送的消息，将消息生产者推送的消息存储到持久化存储区中，以便将所述消息持久化地存储在所述持久化存储区中。
步骤602：第一缓冲区将已存储到所述持久化存储区的消息发送给第二缓冲区。
步骤603a：备缓冲区检查自身是否存在内存溢出，如果存在内存溢出，进入步骤606，如果不存在内存溢出，进入步骤603。
步骤603：备缓冲区缓存第一缓冲区发送的消息。
步骤604：主缓冲区将位于主缓冲区的消息传递给消息消费者。
步骤605a：在将位于所述主缓冲区的消息全部传递给消息消费者之后，主缓冲区检查备缓冲区的当前状态是否为正常状态，如果为非正常状态，进入步骤607，如果为正常状态，进入步骤605。
步骤605：主缓冲区与备缓冲区进行主备切换，结束流程。
步骤606：备缓冲区丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态，结束流程。
步骤607：主缓冲区不与备缓冲区进行主备切换，而从持久化存储区中读取消息，以便将读取的消息传递给消息消费者。
步骤608：主缓冲区判断在备缓冲区中是否存在下一个读取的消息，如果存在，进入步骤609，如果不存在，进入步骤610。
步骤609：将所述备缓冲区的当前状态标记为正常状态，结束流程。
步骤610：维持所述备缓冲区的当前状态，结束流程。
在本申请的另一个可选实施方式中，所述持久化存储区位于所述服务器外部的分布式文件系统中。
在本申请的另一个可选实施方式中，所述主缓冲区和备缓冲区为同构的缓冲区。
在本申请的另一个可选实施方式中，所述消息生产者和消息消费者为运行在所述服务器上的两个进程或者为运行在所述服务器上的一个进程中的两个线程。
由上述实施例可以看出，与现有技术相比，本申请的优点在于：
在消息生产者和消息消费者所在的服务器的内存中设置用于缓存消息的缓冲区，在正常情况下，通过内存访问完成消息传输，保证了消息在消息队列系统中传输的实时性。并且，通过主、备缓冲区的双结构设计，可以实现在将消息生产者推送的消息存储在备 缓冲区的同时，将消息从主缓冲区中传递给消息消费者，更进一步提高了消息在消息队列系统中传输的实时性。另外，还将消息生产者推送的消息存储在持久化存储区中，保证了消息的可靠性。
此外，当消息队列系统中的备缓冲区出现内存溢出问题时，即，在非正常情况下，也可以从持久化存储区中取回内存中丢失的消息，避免了由于消息队列系统中的内存发生内存溢出而导致溢出的消息无法正常地被传递给消息消费者的问题。
所述领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述到的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，可以采用软件功能单元的形式实现。
需要说明的是，本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)或随机存储记忆体(Random Access Memory，RAM)等。
以上对本申请所提供的消息队列系统以及实现消息通信的方法进行了详细介绍，本文中应用了具体实施例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本 申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。

Claims (12)

  1. 一种消息队列系统，其特征在于，包括：第一缓冲区、第二缓冲区以及持久化存储区；其中，所述第一缓冲区和第二缓冲区位于消息生产者和消息消费者所在的服务器的内存中，所述第二缓冲区包括主缓冲区和备缓冲区；
    所述第一缓冲区，用于缓存消息生产者推送的消息，将消息生产者推送的消息存储到所述持久化存储区中，并将已存储到所述持久化存储区的消息发送给所述第二缓冲区；
    所述持久化存储区，用于通过持久化存储介质对所述第一缓冲区发送的消息进行持久化存储；
    所述备缓冲区，用于缓存所述第一缓冲区发送的消息；
    所述主缓冲区，用于将位于所述主缓冲区的消息传递给消息消费者，在将位于所述主缓冲区的消息全部传递给消息消费者之后，与所述备缓冲区进行主备切换。
  2. 根据权利要求1所述的系统，其特征在于，
    所述备缓冲区还用于，在缓存所述第一缓冲区发送的消息之前，检查自身是否存在内存溢出，如果存在内存溢出，丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态，如果不存在内存溢出，继续缓存所述第一缓冲区发送的消息；
    所述主缓冲区还用于，在与所述备缓冲区进行主备切换之前，检查所述备缓冲区的当前状态是否为正常状态，如果为非正常状态，不与所述备缓冲区进行主备切换，而从所述持久化存储区中读取消息，并将读取的消息传递给消息消费者，如果为正常状态，继续与所述备缓冲区进行主备切换。
  3. 根据权利要求2所述的系统，其特征在于，所述主缓冲区还用于，在从所述持久化存储区中读取消息之后，判断在所述备缓冲区中是否存在下一个读取的消息，如果存在，将所述备缓冲区的当前状态标记为正常状态，如果不存在，维持所述备缓冲区的当前状态。
  4. 根据权利要求1至3中任意一项所述的系统，其特征在于，所述持久化存储区位于所述服务器外部的分布式文件系统中。
  5. 根据权利要求1至4中任意一项所述的系统，其特征在于，所述主缓冲区和备缓冲区为同构的缓冲区。
  6. 根据权利要求1至4中任意一项所述的系统，其特征在于，所述消息生产者和消息消费者为运行在所述服务器上的两个进程或者为运行在所述服务器上的一个进程 中的两个线程。
  7. 一种实现消息通信的方法，其特征在于，预先在消息生产者和消息消费者所在的服务器的内存中建立第一缓冲区和第二缓冲区，所述第二缓冲区包括主缓冲区和备缓冲区，所述方法包括：
    第一缓冲区缓存消息生产者推送的消息，将消息生产者推送的消息存储到持久化存储区中，以便将所述消息持久化地存储在所述持久化存储区中；
    第一缓冲区将已存储到所述持久化存储区的消息发送给第二缓冲区；
    备缓冲区缓存第一缓冲区发送的消息；
    主缓冲区将位于主缓冲区的消息传递给消息消费者，在将位于所述主缓冲区的消息全部传递给消息消费者之后，与备缓冲区进行主备切换。
  8. 根据权利要求7所述的方法，其特征在于，所述方法还包括：
    在缓存第一缓冲区发送的消息之前，备缓冲区检查自身是否存在内存溢出，如果存在内存溢出，丢弃所述第一缓冲区发送的消息，并将自身的当前状态标记为非正常状态；
    在与备缓冲区进行主备切换之前，主缓冲区检查备缓冲区的当前状态是否为正常状态，如果为非正常状态，不与备缓冲区进行主备切换，而从持久化存储区中读取消息，以便将读取的消息传递给消息消费者；
    则所述备缓冲区缓存第一缓冲区发送的消息为：
    如果不存在内存溢出，备缓冲区缓存第一缓冲区发送的消息；
    则所述主缓冲区与备缓冲区进行主备切换为：
    如果为正常状态，主缓冲区与备缓冲区进行主备切换。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    在从持久化存储区中读取消息之后，主缓冲区判断在备缓冲区中是否存在下一个读取的消息，如果存在，将所述备缓冲区的当前状态标记为正常状态，如果不存在，维持所述备缓冲区的当前状态。
  10. 根据权利要求7至9中任意一项所述的方法，其特征在于，所述持久化存储区位于所述服务器外部的分布式文件系统中。
  11. 根据权利要求7至10中任意一项所述的方法，其特征在于，所述主缓冲区和备缓冲区为同构的缓冲区。
  12. 根据权利要求7至10中任意一项所述的方法，其特征在于，所述消息生产者和消息消费者为运行在所述服务器上的两个进程或者为运行在所述服务器上的一个进 程中的两个线程。
PCT/CN2016/089308 2015-07-14 2016-07-08 消息队列系统以及实现消息通信的方法 Ceased WO2017008687A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510413284.1A CN107038075B (zh) 2015-07-14 2015-07-14 消息队列系统以及实现消息通信的方法
CN201510413284.1 2015-07-14

Publications (1)

Publication Number Publication Date
WO2017008687A1 true WO2017008687A1 (zh) 2017-01-19

Family

ID=57756860

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/089308 Ceased WO2017008687A1 (zh) 2015-07-14 2016-07-08 消息队列系统以及实现消息通信的方法

Country Status (2)

Country Link
CN (1) CN107038075B (zh)
WO (1) WO2017008687A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109688200A (zh) * 2018-11-30 2019-04-26 北京奇艺世纪科技有限公司 一种消息处理方法、装置及设备
US20230207087A1 (en) * 2020-05-29 2023-06-29 Beijing Jingdong Tuoxian Technology Co., Ltd Prescription order distribution method, system, device and storage medium

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108052296B (zh) * 2017-12-30 2021-02-19 惠龙易通国际物流股份有限公司 一种数据读取方法、设备及计算机存储介质
CN110348826B (zh) * 2018-04-08 2024-05-10 财付通支付科技有限公司 异地多活容灾方法、系统、设备及可读存储介质
CN111538600B (zh) * 2020-02-25 2023-09-12 远景智能国际私人投资有限公司 消息处理方法、装置、计算机设备及存储介质
CN112162875B (zh) * 2020-10-12 2024-08-02 上交所技术有限责任公司 一种交易系统内部高可靠消息传输方法
CN113296976A (zh) * 2021-02-10 2021-08-24 阿里巴巴集团控股有限公司 消息处理方法、装置、电子设备、存储介质及程序产品
CN113296977B (zh) * 2021-02-24 2023-04-07 阿里巴巴集团控股有限公司 一种消息处理方法及装置
CN115629878B (zh) * 2022-10-20 2024-07-19 北京力控元通科技有限公司 一种基于内存交换的数据处理方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1585378A (zh) * 2004-05-24 2005-02-23 中兴通讯股份有限公司 一种交换系统及其中接续消息的处理方法
US7277990B2 (en) * 2004-09-30 2007-10-02 Sanjeev Jain Method and apparatus providing efficient queue descriptor memory access
CN101853238A (zh) * 2010-06-01 2010-10-06 华为技术有限公司 通信处理器间消息通信方法和系统
CN103064749A (zh) * 2013-01-09 2013-04-24 上海斐讯数据通信技术有限公司 一种进程间通信方法
CN103078921A (zh) * 2012-12-28 2013-05-01 Tcl集团股份有限公司 一种发送、接收消息的方法及装置
CN103390049A (zh) * 2013-07-23 2013-11-13 南京联创科技集团股份有限公司 一种基于内存数据库缓存的高速消息队列溢出的处理方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101763289B (zh) * 2009-09-25 2013-11-20 中国人民解放军国防科学技术大学 一种基于共享内存的消息传递方法
US8463802B2 (en) * 2010-08-19 2013-06-11 Sandisk Il Ltd. Card-based management of discardable files
CN102122256B (zh) * 2011-03-28 2013-01-09 中国人民解放军国防科学技术大学 一种用于进程间通信的管道式通信方法及系统
CN103309767B (zh) * 2012-03-08 2016-01-13 阿里巴巴集团控股有限公司 一种客户端日志处理方法和装置
CN103064731A (zh) * 2012-12-26 2013-04-24 人民搜索网络股份公司 一种提高消息队列系统性能的装置及其方法
CN104077405B (zh) * 2014-07-08 2018-06-08 国家电网公司 时序类型数据存取方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1585378A (zh) * 2004-05-24 2005-02-23 中兴通讯股份有限公司 一种交换系统及其中接续消息的处理方法
US7277990B2 (en) * 2004-09-30 2007-10-02 Sanjeev Jain Method and apparatus providing efficient queue descriptor memory access
CN101853238A (zh) * 2010-06-01 2010-10-06 华为技术有限公司 通信处理器间消息通信方法和系统
CN103078921A (zh) * 2012-12-28 2013-05-01 Tcl集团股份有限公司 一种发送、接收消息的方法及装置
CN103064749A (zh) * 2013-01-09 2013-04-24 上海斐讯数据通信技术有限公司 一种进程间通信方法
CN103390049A (zh) * 2013-07-23 2013-11-13 南京联创科技集团股份有限公司 一种基于内存数据库缓存的高速消息队列溢出的处理方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109688200A (zh) * 2018-11-30 2019-04-26 北京奇艺世纪科技有限公司 一种消息处理方法、装置及设备
US20230207087A1 (en) * 2020-05-29 2023-06-29 Beijing Jingdong Tuoxian Technology Co., Ltd Prescription order distribution method, system, device and storage medium

Also Published As

Publication number Publication date
CN107038075A (zh) 2017-08-11
CN107038075B (zh) 2020-12-25

Similar Documents

Publication Publication Date Title
WO2017008687A1 (zh) 消息队列系统以及实现消息通信的方法
CN103207841B (zh) 基于键值对缓存的数据读写方法及装置
US9401958B2 (en) Method, apparatus, and system for migrating user service
CN106170948B (zh) 一种用于双活数据中心的仲裁方法、装置及系统
US7974186B2 (en) Connection recovery device, method and computer-readable medium storing therein processing program
JP5714571B2 (ja) キャッシュクラスタを構成可能モードで用いるキャッシュデータ処理
CN113179327B (zh) 基于大容量内存的高并发协议栈卸载方法、设备、介质
US10162775B2 (en) System and method for efficient cross-controller request handling in active/active storage systems
CN105338090A (zh) 一种基于WebSocket的断点续传方法及装置
CN107948318A (zh) 多节点间的缓存同步方法和系统
WO2017101731A1 (zh) 数据库的服务提供方法和系统
CN103442091B (zh) 一种数据传输方法及装置
CN115129521B (zh) Redis集群间数据同步方法、装置和系统
CN102325171A (zh) 一种监控系统中数据的存储方法及其系统
CN103117955B (zh) 消息传输方法及装置、系统
CN111935490B (zh) 一种直播录流容灾处理方法及系统
CN108667682B (zh) 基于安全网关深度包检测的连接同步方法、装置及介质
JP2009217765A (ja) 複数宛先への同期送信方法、その実施システム及び処理プログラム
CN105389127B (zh) 在存储系统中传输消息的方法、装置及存储系统、控制器
CN113132758B (zh) 内容分发网络的控制方法、装置及计算机程序产品
CN105491101B (zh) 数据的处理方法和装置
CN120909528B (zh) 一种数据写入方法、电子设备
US20230319134A1 (en) Request delivery device, request delivery method, and request delivery program
CN116541204A (zh) 一种基于流式复制的bmc数据同步的方法和装置
CN102547809B (zh) 一种数据发送方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16823834

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16823834

Country of ref document: EP

Kind code of ref document: A1