CN102955717A - Message management equipment and method in distributed message processing system - Google Patents

Abstract

The invention discloses a message management device and method in a distributed message processing system. The message management method includes: receiving a message processing request sent by a message processing device, where the message processing request includes at least the subject of the message to be processed; querying storage partition information corresponding to the subject in the message processing request, and storing the message Send the queried storage partition information to the corresponding message processing device; and process the message location query request from the message processing device, and query the location information of the storage partition corresponding to the storage partition in the message location query request to be processed currently, and send it to the corresponding message processing device; and process the message location update request from the message processing device, based on the message size in the message location update request, update the storage partition corresponding to the storage partition in the message location update request The location information of the current message to be processed.

Description

Message management device and method in distributed message processing system

technical field

The invention relates to the field of message processing, in particular to a message management device and method in a distributed message processing system.

Background technique

There are many message systems currently used for message (such as log) processing, and the more popular one is the distributed message system. There are many specific forms of distributed message systems, such as Spread and Kafka.

The following only uses Kafka, a commonly used distributed message system, as an example for illustration.

Kafka is a distributed message system for log processing, mainly for businesses with large log data capacity but low reliability requirements. The log data in these systems mainly includes user behavior, such as login, browsing, click, share, and like, etc., and also includes system operation logs, such as CPU (Central Processing Unit, central processing unit), memory, disk, network, system and the running status of the process, etc.

FIG. 1 is a schematic diagram of the Kafka system architecture in the related art. The Kafka system includes the following four roles: Producer (message producer, referred to as PD) 11, Broker (abbreviated as BK) 12, Consumer (message processor, referred to as CS) 13 and Zookeeper (manager, referred to as ZK) 14, each There can be multiple roles. Consumer 13 is responsible for processing messages in the Kafka system; Producer 11 is responsible for generating messages in the Kafka system; Broker 12 is a place for storing messages in the Kafka system, and is responsible for receiving messages from Producer 11. When Consumer 13 has a request, these The message is returned to Consumer13; ZooKeeper 14 is an open source distributed application coordination service, which includes a simple set of primitives and is an important component of Hadoop and Hbase. Distributed applications can use it to implement such as: unified naming service, configuration Management, distributed lock service, cluster management and other functions. Producer 11 sends a message to Broker 12, and the message is persistently stored on Broker 12, and Consumer 13 obtains the message from Broker 12 for processing. Zookeeper 14 is used to store some status information of Producer 11, Consumer 13, and Broker 12.

The Kafka system processes messages in a pull (pull) manner, and Consumer 13 obtains messages to be processed from Broker 12 each time. The position Offset of successfully processed messages is saved by Consumer13.

In the above message processing method, Consumer 13 needs to manage and save the position Offset of successfully processed messages by itself. Offset is generally stored in the memory or local disk of the message processing device to which Consumer 13 belongs. When the message processing device terminates unexpectedly or the machine After the disk is broken, the previously saved Offset is lost. When the message processing device is restarted, it will consume from the last saved Offset or from the beginning, and Consumer 13 cannot continue to process the message immediately after the last successfully processed message.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a message management device and method in a distributed message processing system that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, a message management device in a distributed message processing system is provided, and the distributed message processing system further includes one or more message storage devices and one or more message processing devices, each of which The message has a corresponding topic, and the message storage device establishes one or more storage partitions for the corresponding topic, and stores the message in one of the storage partitions of the corresponding topic according to the topic of the message. The message management device includes: a message distribution state storage, configured In order to store at least information about storage partitions in each message storage device and location information about messages currently to be processed by the message processing device in each storage partition; the transmission interface is configured to receive a message processing request sent by the message processing device, The message processing request at least includes the topic of the requested processing message; the partition status queryer is configured to query the storage partition information corresponding to the topic in the message processing request in the message distribution state storage, and send the query information via the transmission interface The storage partition information is sent to the corresponding message processing device; and the message location manager is configured to process the message location query request received from the message processing device through the transmission interface, and obtain the information contained in the message location query request from the message distribution state storage. The location information of the storage partition corresponding to the storage partition currently to be processed is sent to the corresponding message processing device via the transmission interface; and is configured to process the message location update request received through the transmission interface from the message processing device, based on The message size in the message location update request updates the location information of the message currently to be processed in the storage partition corresponding to the storage partition in the message location update request in the message distribution state memory.

Optionally, the message distribution state memory also stores the size of the message to be processed by the message processing device each time; the partition state queryer is also configured to query the message distribution state memory for the size of the message to be processed by the corresponding message processing device, And send it to the corresponding message processing device through the transmission interface.

According to another aspect of the present invention, there is also provided a message management method, which is suitable for execution on a message management device in a distributed message processing system, and the distributed message processing system further includes one or more message storage devices and one or more message processing devices, wherein each message has a corresponding topic, the message storage device establishes one or more storage partitions for the corresponding topic, and stores the message in one of the storage partitions of the corresponding topic according to the topic of the message, The message management method includes: receiving a message processing request sent by a message processing device, where the message processing request includes at least the subject of the message to be processed; querying storage partition information corresponding to the topic in the message processing request, and storing the queried storage partition The information is sent to the corresponding message processing device, wherein, the message management device at least stores the information about the storage partition in each message storage device, and the location information of the message currently to be processed by the message processing device in each storage partition; and processes information from The message location query request of the message processing device, querying the location information of the current message to be processed in the storage partition corresponding to the storage partition in the message location query request, and sending it to the corresponding message processing device; and processing the message from the message processing device The location update request, based on the message size in the message location update request, updates the current location information of the message to be processed in the storage partition corresponding to the storage partition in the message location update request.

Optionally, the above message management method further includes: querying the size of the message to be processed by the corresponding message processing device and sending it to the corresponding message processing device, wherein the message management device also stores the size of the message to be processed by the message processing device each time .

In the technical solution of the present invention, the message management device stores information about the storage partitions in each message storage device and the location information of the messages currently to be processed by the message processing device in each storage partition. When receiving a message, you only need to query the storage partition information on the message storage device for the message under the topic processed by the message processing device from the message management device, and the location information of the message currently to be processed by the message processing device in the corresponding storage partition, that is, A request message can be sent to obtain a message from the message storage device for processing. In this way, even if the message processing device fails, after the message processing device returns to normal, it can still obtain the message for processing at the position where the message was successfully processed last time, avoiding message corruption. Duplicate processing or omission.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:

Fig. 1 shows a schematic diagram of a Kafka system architecture;

Fig. 2 shows a schematic diagram of a distributed message processing system according to an embodiment of the present invention;

Fig. 3 shows a schematic diagram of an internal storage structure of a message storage device according to an embodiment of the present invention;

Fig. 4 shows the flowchart of the message management method according to one embodiment of the present invention; And

Fig. 5 shows a flowchart of a message processing method according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

Fig. 2 shows a schematic diagram of a distributed message processing system according to an embodiment of the present invention. As shown in the figure, the distributed message processing system includes a message generating device 100 , a message storage device 200 , a message processing device 300 and a message management device 400 . There may be one or more message generating devices 100, message storage devices 200, and message processing devices 300 respectively, and only two message generating devices 100, two message storage devices 200, and one message processing device are shown in FIG. 300 and one message management device 400, those skilled in the art can understand that the embodiment of the present invention does not limit the number of message generation device 100, message storage device 200, message processing device 300 and message management device 400.

The message generating device 100 may be a network device, a server, a computer or any other device capable of generating messages, where the messages may be any kind of information data, such as log data. Each message has a corresponding topic. For example, the topic can be a specific user behavior, such as login behavior, browsing behavior, click behavior, sharing behavior, or liking behavior, and the topic can be a specific system operation log. , such as CPU, memory, disk, network, system and process running status, etc. For example, in one embodiment, the message generating device 100 is a web server. When a user browses a web page of the web server, the web server generates a message with the subject of browsing the web page.

After the message generation device 100 generates a message, it sends the generated message to the message storage device 200 . When the message generation device 100 sends a message to the message storage device 200, it may send a message randomly, or may send a message according to a callback function specified by the user. An example code for the message generating device 100 to send the generated message to the message storage device 200 is as follows:

$zkCluster = 'localhost:2181';

$msg=array 'message content');

$topic = 'test';

$Producer = new Kafka_Producer($zkCluster);

$Producer->send(array($msg), $topic');

Those skilled in the art can understand that the above message transfer code is just an example, and any code implementation that can realize sending the message generated by the message generating device 100 to the message storage device 200 is within the protection scope of the present invention.

After receiving the message sent by the message generating device 100, the message storage device 200 stores the received message. The message storage device 200 has multiple specific storage methods. For example, the message storage device 200 stores messages according to the subject of the message. Specifically, one or more storage partitions may be established for each message topic, and each received message is stored in one of the storage partitions of the corresponding topic.

In order to better introduce the internal storage structure of the message storage device 200, please also refer to FIG. 3, which is a schematic diagram of the internal storage structure of the message storage device 200 according to an embodiment of the present invention. In Fig. 3, the first message storage device 2001, the second message storage device 2002 and the third message storage device 2003, and the message storage path inside each message storage device are shown. In addition, the first message generating device 1001 and the second message generating device 1002 that perform information interaction with these message storage devices, and the first message processing device 3001 and the second message processing device 3002 are also shown.

The topics generated by the first message generating device 1001 and the second message generating device 1002 are Topic1 and Topic2 messages, which will be sent to the first message storage device 2001 and the second message storage device which can be used to store the messages of Topic1 and Topic2. device 2002 and a third message storage device 2003.

Because the distributed message processing system performs message management based on the topic (Topic) of the message, correspondingly, the message storage device also stores the message based on the Topic. The messages of each Topic can be stored in one or more storage partitions (Partition). When stored in multiple storage partitions, these multiple storage partitions can be located on one message storage device or on multiple message storage devices. . In Fig. 3, the subject is Topic1 message and is stored in storage partition part1 and storage partition part2 under Topic1 in the first message storage device 2001, storage partition part1 and storage partition part2 under Topic1 in the second message storage device 2002, and the first message storage device 2002. Three storage partitions part1 under Topic1 in the message storage device 2003. Messages with topic Top2 are stored in the storage partition part1 under Topic2 in the first message storage device 2001 , the storage partition part1 under Topic2 in the second message storage device 2002 , and the storage partition part1 under Topic2 in the third message storage device 2003 . Each bucket has multiple messages, and each message has a different position relative to the header of the bucket. For example, according to an embodiment of the present invention, each message has a message id, and the message id is determined by its logical location, that is, the storage location of the message can be directly located from the message id, avoiding additional mapping from the id to the storage location.

An internal storage structure of the message storage device 200 in FIG. 2 has been introduced in detail through FIG. 3 above, and the following will return to the embodiment shown in FIG. 2 . After the message storage device 200 stores the received messages, it can send the information about each update of the storage partition to the message management device 400 for storage by the message management device 400 .

The message generation device 100 and the message storage device 200 in an embodiment have been introduced above, and the message processing device 300 and the message management device 400 will continue to be introduced below in conjunction with the specific processing of messages under a certain topic. Since each component in the message processing device 300 and each component in the message management device 400 interact with each other more information, the following first briefly introduces each component included in the message processing device 300 and each component included in the message management device 400, and then In the process of specific information interaction, the various components involved will be introduced in more detail.

The message processing device 300 includes a transmission interface 304, a storage information acquirer 302, a message location manager 306, and a message acquirer 308, and the message management device 400 includes a transmission interface 404, a message distribution state storage 402, a partition status queryer 406, and a message location management device 408.

When the message processing device 300 needs to process the message of a certain Topic, it first generates a message processing request through the storage information acquirer 302 of the message processing device 300, and the message processing request includes at least the subject of the message processing device 300 requested to process the message, and then passes The transmission interface 304 of the message processing device 300 transmits the message processing request to the transmission interface 404 of the message management device 400 .

After the transmission interface 404 of the message management device 400 receives the message processing request from the message processing device 300, it sends the message processing request to the partition state queryer 406, and then the partition state queryer 406 queries the message distribution state memory 402 related to the message processing request. One or more bucket information corresponding to the topic in the message processing request.

Specifically, since the message distribution state memory 402 stores information about storage partitions received by the message management device 400 and from each message storage device 200, the distribution state queryer 406 can process the message according to the information specified in the message processing request. For a topic, query from the message distribution state storage 402 which or which storage partitions of the message storage device 200 the messages under the topic are stored on. For example, it is assumed that the distributed status queryer 406 inquires a total of 5 storage partitions to store messages under the topic specified in the message processing request, and each storage partition is identified by broker_id-partition_id, which are: b1-p1, b1-p2, b2-p1 , b2-p2, b3-p1, the queried storage partition information including the five storage partition identifiers is sent to the corresponding transmission interface 304 of the message processing device 300 through the transmission interface 404 .

After the transmission interface 304 of the message processing device 300 receives one or more storage partition information corresponding to the topic in the message processing request from the message management device 400, it forwards it to the message location manager 306 for processing.

Specifically, the message location manager 306 sequentially selects one storage partition information from one or more storage partition information, and then acquires from the message management device 400 the location information of the currently processed message corresponding to the selected storage partition information. For example, the message location manager 306 includes a selection module and a message location query module. First, the selection module sequentially selects a storage partition information from one or more storage partition information. When selecting storage partition information, it may be from one or more storage partition information. In the partition information, a storage partition information is randomly selected sequentially, or a storage partition information can be sequentially selected according to the number of the message storage device 200 and the number of the storage partition on the corresponding message storage device 200 . Then the message location query module generates a corresponding message location query request according to the storage partition information selected by the selection module, the message location query request includes the storage partition information selected by the selection module, and then sends the message generated by the message location query module The location query request is sent to the transmission interface 404 of the message management device 400 so as to acquire the location information of the message currently to be processed corresponding to the storage partition information from the message management device 400 .

After receiving the message location query request from the message processing device 300 , the transmission interface 404 of the message management device 400 forwards the message location query request to the message location manager 408 . Because the position information of the message to be processed by the relevant message processing device in each storage partition is also stored in the message distribution state memory 402, so after the message position manager 408 receives the message position query request, the message from the message distribution state memory 402 Query the location information of the message currently to be processed in the storage partition corresponding to the storage partition in the message location query request, and send the query result to the transmission interface 304 of the corresponding message processing device 300 through the transmission interface 404 after the query is completed. If there is no location information of the message to be processed in the storage partition corresponding to the storage partition in the message location query request in the message distribution status memory 402, the message location manager 408 can send the message to be processed currently in the corresponding storage partition The initial value of the location information of is set to 0, and is sent to the corresponding message processing device 300. In addition, the message location manager 408 may also directly send the location information of the message to be processed by the corresponding storage partition to the message processing device 300 directly, and the message processing device 300 will send the location information of the message to be processed by the corresponding storage partition to The initial value is set to 0.

It should be noted that, in the previous embodiment, the stored information acquirer 302 of the message processing device 300 first sends a message processing request to the message management device 400, and after obtaining the result returned by the message management device 400, the message location manager 306 then sends a message processing request to the message management device 400. The message management device 400 sends a message location query request. In yet another embodiment, the stored information acquirer 302 and the message location manager 306 of the message processing device 300 may also send the message processing request and the message location query request to the message management device 400 at the same time. For example, assume that the message management device 400 receives a message processing request and a message location query request from the message processing device 300 at the same time, and then the distribution status queryer 406 queries a total of 5 storage partitions according to the message processing request to store information under the specified topic in the message processing request. message, then the message location manager 408 further inquires the location information of the messages currently to be processed by the five storage partitions according to the query result of the distribution status queryer 406 and the message location query request, and finally lists the current location information of the five storage partitions respectively The location information of the message to be processed is sent to the corresponding message processing device 300 together.

After the transmission interface 304 of the message processing device 300 receives the location information of the message currently to be processed in the storage partition corresponding to the storage partition in the message location query request returned by the message management device 400, the location information is passed through the message location The manager 306 forwards it to the message acquirer 308; or the transmission interface 304 may directly send the received location information to the message acquirer 308.

After the message processing device 300 inquires from the message management device 400 the storage partition information of the message of the processed topic and the current location information of the message to be processed in the corresponding storage partition, the message obtainer 308 can generate a message obtaining request, and transmit The interface 304 sends the message acquisition request to the message storage device 200 where the storage partition selected by the message manager 306 is located. The four parameters of Topic, Partition, Offset, and Fetchsize need to be specified in the message acquisition request. Topic refers to the message of the topic to be processed, Partition refers to the storage partition under the topic to be processed, and Offset refers to the message from the storage partition. The location starts to fetch messages, and Fetchsize refers to the length of messages fetched from the Offset of the storage partition. Among these four parameters, the Topic, Partition and Offset parameters have been obtained in the interaction process with the message management device 400 above, and the Fetchsize parameter can be determined by the message acquirer 308 according to the status of the message processing device 300 itself.

An example code for the message acquirer 308 of the message processing device 300 to send the generated message acquisition request to the corresponding message storage device 200 is as follows:

$zkCluster = 'localhost:2181';

$topic = 'test';

$Consumer = new Kafka_Consumer($zkCluster, $topic);

$msg=$Consumer->fetchMsg();

Those skilled in the art can understand that the above code for transmitting the message acquisition request is just an example, and any code implementation that can send the message acquisition request generated by the message processing device 300 to the message storage device 200 is within the protection scope of the present invention.

It should be noted that the message size Fetchsize to be processed by the message processing device 300 is generally a fixed value, such as 1M (megabytes). The Fetchsize may be stored locally on the message processing device 300 . In this case, when the message obtainer 308 generates a message obtaining request, it only needs to query the Fetchsize in the local disk of the message processing device 300 . Those skilled in the art can understand that the Fetchsize can also be stored in the message distribution state memory 402 of the message management device 400. In this case, when the partition state queryer 403 inquires information from the distribution state memory 402 to process the message acquisition request, The Fetchsize is acquired and sent to the message location manager 306 of the message processing device 300 together with the information about the storage partition. Of course, the Fetchsize acquisition request can also be generated by the stored information acquirer 302 or the message location manager 306 of the message processing device 300, and then the Fetchsize acquisition request is sent to the message management device 400, and then from the message distribution state memory 402 in the message management device 400 Query to obtain the message size Fetchsize to be processed.

After the message storage device 200 receives the message acquisition request generated by the message acquirer 308 of the message processing device 300, it queries the storage partition corresponding to the Topic and Partition in the message acquisition request, and starts from the storage partition from the position specified by Offset Intercept the message whose length is Fetchsize, and then return the queried message to the message processing device 300 . In yet another embodiment, the message processing device 300 further includes a message processor, and the message processor parses the message returned by the message storage device 200 according to the message format. If the three messages and their lengths are obtained after parsing: msg1, len1; msg2 , 1en2; msg3, len3, then the message processor processes these three messages in turn.

In yet another embodiment, the message processing device 300 further includes a message location updater. After the message processor successfully processes the message, the message location updater generates a message location update request, and sends the generated message location update request to the message management device 400 via the transmission interface 304. The message location update request includes the message processor's latest or The size of the message successfully processed multiple times and the storage partition information where the message is located.

After receiving the message location update request from the message processing device 300, the transmission interface 404 of the message management device 400 forwards it to the message location manager 408. Based on the message size in the message location update request, the message location manager 408 updates the location information of the currently processed message in the storage partition corresponding to the storage partition in the message location update request in the message distribution state storage 402 . There are many ways to implement the update of the location information offset of the message currently to be processed. For example, the message location updater can generate a message location update request containing the size of the processed message every time the message processor successfully processes a message, for example, when the message processor successfully processes the message msg1, the message location updater generates a request containing msg1 The message position update request of the size len1, the message position manager 408 of the message management device 400 stores the new position information offset+len1 into the node /consumer/offset/b1-p1 in the message distribution state memory 402. When the message processor has successfully processed the second message msg2, the message location updater generates a message location update request containing the size len2 of msg2, and the message location manager 408 of the message management device 400 adds len2 to the new offset , and update the content of the node /consumer/offset/b1-p1 in the message distribution state storage 402.

For topics that do not require high security, the message location updater can generate a message location update request to update the corresponding storage partition in the message distribution state memory 402 of the message management device 400 after the message processor has successfully processed multiple messages the offset. For example, in the above example, after the message processor has successfully processed the three messages msg1, msg2, and msg3, the message location updater can generate a message location update request containing three messages of size len1+len2+len3 and send it to the message The management device 400, the message position manager 408 of the message management device 400 stores the new position information offset+len1+len2+len3 into the node /consumer/offset/b1-p1 in the message distribution state memory 402, and updates the offset , which can increase some efficiency.

From the above multiple embodiments of the present invention, it can be seen that the embodiment of the present invention saves the position information offset of the message currently to be processed by the message processing device in each storage partition in the message management device, avoiding the message processing device every When processing a message for the first time, the offset needs to be calculated locally, which can reduce the cost of using message processing equipment and improve the utilization efficiency of its resources.

Further, because the message management device 400 stores information about storage partitions in each message storage device 200 and location information about messages currently to be processed by the message processing device 300 in each storage partition, when the message processing device 300 When acquiring a message to be processed, it is only necessary to inquire from the message management device 400 about the storage partition information on the message storage device 200 of the message under the topic processed by the message processing device 300, and the information currently to be processed by the message processing device 300 in the corresponding storage partition. The location information of the message, that is, send a message acquisition request to obtain the corresponding message from the message storage device 200 for processing. In this way, even if the message processing device 300 fails, after the message processing device 300 returns to normal, it can still succeed the last time. The location where the message is processed obtains the message for processing, avoiding repeated processing or omission of the message.

Corresponding to the aforementioned distributed message processing system according to an embodiment of the present invention, FIG. 4 shows a flow chart of a message management method adapted to be executed on a message management device in a distributed message processing system according to an embodiment of the present invention. The distributed message processing system includes one or more message generating devices 100 as described in the embodiment of Figure 2, one or more message storage devices 200 as described in the embodiment of Figure 2, one or more The described message processing device 300 and the message management device 400 as described in the embodiment of FIG. 2 . The message management method is suitable for execution on the message management device 400 described in the embodiment of FIG. 2 .

The message management method starts at step S410. In step S410, a message processing request sent by a message processing device is received, and the message processing request includes at least the subject of the message to be processed. This step can be performed through the transmission interface 404 in the aforementioned message management device 400 , and related technical implementation can refer to the relevant descriptions of the transmission interface 404 in the aforementioned message management device 400 in various embodiments, which will not be repeated here.

After the message processing request is received in step S410, step S420 queries the message management device for storage partition information corresponding to the topic in the message processing request, and sends the queried storage partition information to the corresponding message processing device. Specifically, since information about storage partitions in each message storage device and location information about messages to be processed by the message processing device in each storage partition are usually stored in the message management device, in this step, According to the subject of the message specified in the message processing request, it may be queried from the message management device which storage partition(s) of the message storage device(s) the message under the subject is stored in. For example, assume that there are 5 storage partitions to store messages of this topic, and each storage partition is identified by broker_id-partition_id, which are: b1-p1, b1-p2, b2-p1, b2-p2, b3-p1, then Send the queried storage partition information including the identifiers of the five storage partitions to the corresponding message processing device. When the message management device also stores the size of the message to be processed by the message processing device, it can also inquire about the size of the message to be processed by the corresponding message processing device while querying the relevant storage partition information, and send it to the corresponding message processing device. equipment. This step can be performed by the partition state queryer 406 in the aforementioned message management device 400. For related technical implementations, reference can be made to the relevant descriptions of the aforementioned partition status queryer 406 in various embodiments, which will not be repeated here.

Through step S420, the message processing device has obtained the storage partition information of the messages under the topic to be processed, so the subsequent message processing device will generate a message location query request according to the selected storage partition information, and send it to the message management device. Then, in step S430, the message location query request from the message processing device is processed. For example, the message management device queries the location information of the message currently to be processed in the storage partition corresponding to the storage partition in the message location query request, and then sends the queried location information to the corresponding message processing device.

There are many ways to implement this step. For example, suppose there are five storage partitions obtained by the message processing device, namely: b1-p1, b1-p2, b2-p1, b2-p2, b3-p1, in order Select a storage partition from these 5 storage partitions in turn to generate a message location query request. If you want to query the location information of the message currently processed by the b1-p1 storage partition, generate a message location query request containing the b1-p1 storage partition to In step S430, the message management device inquires from the corresponding node /consumer/offset/b1-p1 in the message management device the location information offset of the message currently to be processed in the b1-p1 storage partition, and sends it to the corresponding message processing device. If the node does not exist in the message management device, set the initial value of the position information of the message to be processed in the b1-p1 storage partition to 0, and send it to the corresponding message processing device. In addition, the information that the node does not exist may also be directly sent to the message processing device, and the message processing device sets the initial value of the message position information to be processed in the b1-p1 storage partition to 0. This step can be performed by the message location manager 408 in the aforementioned message management device 400. For related technical implementations, reference can be made to the related descriptions of the aforementioned message location manager 408 in various embodiments, which will not be repeated here.

Through step S430, the message processing device has obtained the location information of the message currently to be processed in the selected storage partition, so it can obtain the corresponding message from the corresponding storage partition of the message storage device accordingly, and process the message. Since the processing of the message by the message processing device will result in a change of the location information of the subsequent processed message, the message processing device will send a message location update request to the message management device. Specifically, in step S440, the message location update request from the message processing device is processed, and based on the message size in the message location update request, the storage partition corresponding to the storage partition in the message location update request in the message management device is updated The position information of the current message to be processed. When updating the current message position information offset to be processed in the corresponding storage partition in the message management device, the message processing device may send a message position update request to update the offset of the corresponding storage partition in the message management device every time a message is successfully processed. For example, when the message processing device successfully processes the message msg1, the message management device stores the new location information offset+len1 into the corresponding node /consumer/offset of the message management device according to the message location update request containing len1 sent by the message processing device /b1-p1; when the message processing device successfully processes the second message msg2, the message management device adds len2 to the new offset according to the message position update request containing len2 sent by the message processing device, and updates the corresponding Content in the message management device.

For topics that do not require high security, you can update the offset in the corresponding message management device after the message processing device successfully processes multiple messages. For example, in the above example, you can also update the offset after the message processing device successfully processes After three messages, the message management device stores the new location information offset+len1+len2+len3 into the corresponding node /consumer/offset of the message management device according to the message location update request containing len1+len2+len3 sent by the message processing device /b1-p1, this can increase some efficiency.

Step S430 can be executed by the message location manager 408 in the aforementioned message management device 400. For related technical implementation, reference can be made to the related descriptions of the aforementioned message location manager 408 in various embodiments, which will not be repeated here.

In the above embodiment, since the information about the storage partition in each message storage device and the location information about the message currently to be processed by the message processing device in each storage partition are stored on the message management device, when the message processing device acquires the When processing a message, it is only necessary to inquire from the message management device the storage partition information of the message under the topic processed by the message processing device on the message storage device, and the location information of the message currently to be processed by the message processing device in the corresponding storage partition, You can send a message acquisition request to obtain the corresponding message from the message storage device for processing. In this way, even if the message processing device fails, after the message processing device returns to normal, you can still obtain the message for processing at the position where the message was successfully processed last time. Repeated processing or omission of messages is avoided.

Corresponding to the aforementioned distributed message processing system according to one embodiment of the present invention, FIG. 5 shows a flow chart of a message processing method adapted to be executed on a message processing device in a distributed message processing system according to one embodiment of the present invention. The distributed message processing system includes one or more message generating devices 100 as described in the embodiment of Figure 2, one or more message storage devices 200 as described in the embodiment of Figure 2, one or more The described message processing device 300 and the message management device 400 as described in the embodiment of FIG. 2 . The message processing method is suitable for execution on the message processing device 300 described in the embodiment of FIG. 2 .

The message processing method starts at step S510. In step S510, a message processing request is generated, and the message processing request includes at least the subject of the message to be processed. Specifically, the message processing device generates a message processing request according to the subject of the message to be processed. This step can be performed by the stored information acquirer 302 in the aforementioned message processing device 300, and related technical implementations can refer to the related descriptions of the aforementioned stored information acquirer 202 in various embodiments, which will not be repeated here.

Furthermore, in step S520, the message processing request generated in step S510 is sent to the message management device, and one or more storage partition information corresponding to the subject in the message processing request sent by the message management device is received. After sending the message processing request to the message management device, the message management device queries information of one or more storage partitions corresponding to the topic according to the topic specified in the message processing request, and sends it to the corresponding message processing device. This step can be performed through the transmission interface 304 in the aforementioned message processing device 300 , and related technical implementation can refer to the relevant descriptions of the aforementioned transmission interface 304 in the message processing device 300 in various embodiments, which will not be repeated here.

Thereafter, in step S530, one storage partition information is selected from the one or more storage partition information obtained in step S520, and then the position information of the currently to-be-processed message corresponding to the selected storage partition information is obtained. Specifically, it may be implemented in the following manner: sequentially select one storage partition information from one or more storage partition information obtained in step S520 (this step may be performed by the aforementioned selection module). For example, one storage partition information may be randomly selected sequentially from one or more storage partition information, or one storage partition information may be sequentially selected according to the number of the message storage device and the number of the storage partition on the corresponding message storage device ; Then generate a message location query request according to the selected storage partition information and send it to the message management device. The message location query request includes the selected storage partition information, so as to obtain the current information corresponding to the selected storage partition information from the message management device. The location information of the message to be processed (this step can be executed through the previous message location query module). Step S530 can be executed by the message location manager 306 in the aforementioned message processing device 300, and related technical implementations can refer to the related descriptions of the aforementioned message location manager 306 in the message processing device 300 in various embodiments, which are not repeated here. repeat.

Then, in step S540, a message acquisition request is generated according to the storage partition information obtained in steps S520 and S530 and the location information of the message currently to be processed in the corresponding storage partition, and the message acquisition request includes the subject of the message to be processed, the selected The storage partition of the storage partition, the location information of the current message to be processed in the storage partition, and the size of the message to be processed. This step can be performed by the message acquirer 308 in the aforementioned message processing device 300, and related technical implementations can refer to the related descriptions of the message acquirer 308 in the aforementioned message processing device 300 in various embodiments, which will not be repeated here. .

Finally, in step S550, the message acquisition request generated in step S540 is sent to the message storage device where the corresponding storage partition is located, and the message corresponding to the message acquisition request returned by the message storage device where the storage partition is located is received. This step can be performed through the transmission interface 304 in the aforementioned message processing device 300 , and related technical implementation can refer to the relevant descriptions of the aforementioned transmission interface 304 in the message processing device 300 in various embodiments, which will not be repeated here.

In addition, it should be pointed out that, when step S520 is executed, the message processing request and the message location query request may also be sent to the message management device at the same time. For example, assuming that the message management device inquires according to the message processing request that there are 5 storage partitions to store messages under a certain topic, the message management device can further query the location information of the messages currently processed by the 5 storage partitions according to the query results , and return the message location information currently to be processed by the five storage partitions to the message processing device at the same time in response to the request of step S520. In this way, step S530 may also sequentially select one storage partition from the five storage partitions and the position information of the message currently to be processed in the corresponding storage partition.

The above message processing method may further include the following step: processing the received message corresponding to the message acquisition request. For example, the corresponding message returned by the message storage device according to the message acquisition request is parsed according to the message format. If three messages and their lengths are obtained after parsing: msg1, len1; msg2, len2; deal with. This step can be performed by the message processor in the aforementioned message processing device 300 , and related technical implementations can refer to the relevant descriptions of the message processor in the aforementioned message processing device 300 in various embodiments, which will not be repeated here.

The above message processing method further includes the following steps: generating a message location update request and sending it to the message management device, the message location update request includes the size of the latest one or more successfully processed messages and the storage partition information where the message is located. In this step, there can be multiple implementations. For example, after each message is successfully processed, a message location update request including the size of the processed message can be sent to the message management device, and the message management device updates the current request of the corresponding storage partition. The position information offset of the processed message. For example, when the message msg1 is successfully processed, a message location update request containing the size len1 of msg1 is generated and sent to the message management device, and the message management device stores the new location information offset+len1 into the corresponding node /consumer/offset/b1 -p1; when the second message msg2 is successfully processed, a message position update request containing msg2 with a size of len2 is generated and sent to the message management device, and the message management device adds len2 to the new offset and updates The content of the corresponding node /consumer/offset/b1-p1.

In another implementation, for a topic that does not require high security, after successfully processing multiple messages, a message location update request can be generated to update the offset of the corresponding storage partition in the message management device, for example, in the above In the example of , after the three messages msg1, msg2, and msg3 are successfully processed, a message location update request containing three messages of size len1+len2+len3 can be generated and sent to the message management device, and the message management device will update the new location information offset+len1+len2+len3 is stored in the corresponding node /consumer/offset/b1-p1, and the offset is updated, which can increase some efficiency. This step can be performed by the message location updater in the aforementioned message processing device 300. For related technical implementation, please refer to the relevant descriptions of the message location updater in the aforementioned message processing device 300 in various embodiments, which will not be repeated here.

In the above embodiment, since the message management device stores the information about the storage partitions in each message storage device and the location information about the messages currently to be processed by the message processing device in each storage partition, when the message processing device obtains the When processing a message, it is only necessary to inquire from the message management device the storage partition information of the message under the topic processed by the message processing device on the message storage device, and the location information of the message currently to be processed by the message processing device in the corresponding storage partition, You can send a message acquisition request to acquire messages from the message storage device for processing. In this way, even if the message processing device fails, after the message processing device returns to normal, you can still obtain the message for processing at the position where the message was successfully processed last time, avoiding Duplicate processing or omission of messages.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other device. Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, in order to streamline the present disclosure and to facilitate understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together into a single embodiment , figure, or description of it. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the device according to the embodiments of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

The application may be applied to computer systems/servers that are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments and/or configurations suitable for use with computer systems/servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, Microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, among others. Computer systems/servers may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server can be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including storage devices.

Claims (4)

1. message management equipment in the distributed message disposal system, this distributed message disposal system also comprises one or more information storaging apparatus and one or more message processing device, wherein every message has corresponding theme, described information storaging apparatus is that one or more partition holding set up in corresponding theme, and according to the theme of message with message stores in one of partition holding of corresponding theme, described message management equipment comprises:

The message distribution status register is configured to store at least in each information storaging apparatus the positional information of the current message to be processed of relevant information treatment facility in the information of relevant partition holding and each partition holding;

Transmission interface is configured to the Message Processing request that the receipt message treatment facility sends, and described Message Processing request comprises the theme of the processing messages of asking at least;

The subregion state requestor, be configured to the inquiry partition holding information corresponding with the theme in the described Message Processing request in described message distribution status register, and via described transmission interface the partition holding information that inquires sent to the corresponding message treatment facility; And

The message location manager, be configured to process receive by described transmission interface, from the message position query requests of message processing device, from described message distribution status register, obtain the current of the partition holding corresponding with the partition holding in this message position query requests and want the positional information of processing messages and send to the corresponding message treatment facility via described transmission interface; And be configured to process receive by described transmission interface, from the message position updating request of message processing device, based on the message size in the described message position updating request, upgrade in the described message distribution status register, the current positional information of wanting processing messages of the partition holding corresponding with the partition holding in the described message position updating request.

2. message management equipment as claimed in claim 1 also stores the size of the each message to be dealt with of message processing device in the wherein said message distribution status register;

Described subregion state requestor also is configured to the size of inquiry corresponding message treatment facility message to be dealt with in described message distribution status register, and sends to the corresponding message treatment facility by described transmission interface.

3. information management method, the message management equipment that the method is suitable in the distributed message disposal system is carried out, this distributed message disposal system also comprises one or more information storaging apparatus and one or more message processing device, wherein every message has corresponding theme, described information storaging apparatus is that one or more partition holding set up in corresponding theme, and according to the theme of message with message stores in one of partition holding of corresponding theme, described information management method comprises:

The Message Processing request that the receipt message treatment facility sends, described Message Processing request comprises the theme of the processing messages of asking at least;

Inquire about the partition holding information corresponding with the theme in the described Message Processing request, and the partition holding information that inquires sent to the corresponding message treatment facility, wherein, store at least in each information storaging apparatus the positional information of the current message to be processed of relevant information treatment facility in the information of relevant partition holding and each partition holding in the described message management equipment; And

Processing is inquired about partition holding current corresponding with the partition holding in this message position query requests and is wanted the positional information of processing messages and send to the corresponding message treatment facility from the message position query requests of message processing device; And processing is from the message position updating request of message processing device, based on the message size in the described message position updating request, upgrade the current positional information of wanting processing messages of the partition holding corresponding with the partition holding in the described message position updating request.

4. information management method as claimed in claim 3 also comprises:

The size of inquiry corresponding message treatment facility message to be dealt with also sends to the corresponding message treatment facility, also stores the size of the each message to be dealt with of message processing device in the wherein said message management equipment.

Images