KR102124897B1 - Distributed Messaging System and Method for Dynamic Partitioning in Distributed Messaging System - Google Patents

Abstract

The present invention discloses a distributed message system and a dynamic partitioning method in a distributed message system.
A dynamic partitioning method in a distributed message system according to an embodiment of the present invention includes determining whether a consumer without a partition assigned among a plurality of consumers exists; Equally dividing the messages remaining in the first partition in half among the plurality of partitions, and creating two new partitions to store the evenly divided messages in the two new partitions; Allocating the two new partitions to a consumer without the assigned partition and to a consumer that has been assigned to the first partition; And updating the metadata including list information about partitions and mapping information between each consumer and a partition allocated to each consumer; It may include.

Description

Distributed messaging system and method for dynamic partitioning in distributed messaging system

The present invention relates to a distributed message system and a message processing method in a distributed message system, and more particularly, to a method for dynamically partitioning a message in a distributed message system and a distributed message system to which the method is applied.

Distributed messaging system (distributed messaging system) is designed to effectively deliver messages in various types of distributed computer systems and platforms, there are currently a number of open source-based solutions. Examples of specific solutions include ActiveMQ, Kafka, RabbitMQ, and the like.

Among them, Apache kafka implements a fully-distributed message system unlike other solutions based on the existing centralized architecture. According to this Apache Kafka, multiple consumers can share messages in a traditional queuing method, and it is also possible to deliver multiple messages in a publish-subcribe form around a specific topic. It is possible.

A fully-distributed message system such as the existing Apache Kafka is configured to access only one consumer for a single message partition to maximize parallelism. In addition, the existing fully distributed message system uses a static partitioning scheme to divide the entire messages into multiple partitions and store them in multiple servers. That is, the existing fully distributed message system divides all messages equally and stores them in each partition. This static partitioning method assumes that there is no difference in processing performance of each consumer, and that only the total message size affects the performance of the entire system. However, there may be a difference in the processing performance of the consumers, and in the case of not only delivering a message, but also a large-task composed of large-scale tasks (Many-Task Computing), the static partitioning method may cause a serious load balancing problem.

In addition, when one message is used as a task for performing an actual operation, there may be a deviation in the time each task is executed. Accordingly, a consumer may end up processing the partition in charge earlier than other consumers. In this case, since the number of consumers is greater than the partitions to be processed as a whole, a fully distributed message system such as Kafka performs redistribution of the entire message partitions through a re-balancing operation. . However, there is a problem in that message processing is impossible while such a redistribution operation occurs. In particular, as the number of consumers newly participating in the message processing task increases or the number of consumers who have completed message processing before other consumers increases, the redistribution task must be performed more or longer, which can seriously affect the overall system performance. have.

The present invention was created by recognizing the problems described above, and provides a fully distributed message system that maximizes parallelism and improves system performance, such as a conventional fully-distributed message system. It is aimed at.

In addition, according to the present invention, when the time required for each message processing is various (eg, MTC (Many-Task Computing) scientific application processing), the message partition is dynamically partitioned with minimal effort in a fully distributed message system. The aim is to provide an algorithm that can improve load balancing and contribute to overall system performance improvement.

In order to achieve the above object, a distributed message system according to an aspect of the present invention is disclosed.

(1) A distributed message system according to an embodiment of the present invention includes: a plurality of partitions that divide and store messages; A consumer group comprising a plurality of consumers-messages stored in one partition are processed by one consumer included in the consumer group; And a message partitioning module that manages metadata including list information for the plurality of partitions and mapping information between each consumer and a partition allocated to each consumer.

(2) In the above (1), the message partitioning module may update the metadata each time a dynamic redistribution operation is performed.

(3) In the above (2), the dynamic redistribution operation may be performed according to a predetermined period or whenever a change occurs in the consumer group.

(4) In the above (3), when a change occurs in the consumer group, when a new consumer joins the consumer group, at least one consumer among the plurality of consumers is stored in a partition assigned to the consumer. It may be any one of cases in which all the messages have been processed, and at least one of the plurality of consumers has failed to process a message stored in a partition allocated to itself.

(5) The process according to any one of (2) to (4), wherein the dynamic redistribution operation is performed by the message partitioning module: determining whether there is a consumer without a partition assigned among the plurality of consumers. ; A process of equally dividing the messages remaining in the first partition among the plurality of partitions in half, creating two new partitions, and storing the evenly divided messages in the two new partitions, respectively; And a process of allocating the two new partitions to a consumer without the assigned partition and a consumer that has been assigned to the first partition. It may include.

(6) In the above (5), the process of generating the meta data before the determining process; may further include.

(7) In any one of (5) and (6) above, the first partition may be a partition in which messages to be processed among the plurality of partitions remain most.

(8) The process according to any one of (5) to (7), after the determining process, comparing the number of messages remaining in the first partition with a threshold value; It may further include.

(9) In any one of (1) to (8), the meta data may further include list information for the plurality of consumers.

In order to achieve the above object, a dynamic partitioning method according to another aspect of the present invention is disclosed.

(10) A dynamic partitioning method according to an embodiment of the present invention includes: determining whether a consumer without a partition assigned among a plurality of consumers exists; Equally dividing the messages remaining in the first partition in half among the plurality of partitions, and creating two new partitions to store the evenly divided messages in the two new partitions, respectively; Allocating the two new partitions to a consumer without the assigned partition and to a consumer that has been assigned to the first partition; And updating the metadata including list information about partitions and mapping information between each consumer and a partition allocated to each consumer; It may include.

(11) In the above (10), prior to the determining step, generating the metadata; may further include.

(12) In any one of (10) and (11) above, the meta data may further include list information for the plurality of consumers.

(13) In any one of (10) to (12), the first partition may be a partition in which messages to be processed among the plurality of partitions remain the most.

(14) The method according to any one of (10) to (13), after the determining step, comparing the number of messages remaining in the first partition with a threshold value; It may further include.

To achieve the above object, a storage medium storing a program programmed to perform a dynamic partitioning method according to another aspect of the present invention is disclosed.

(15) A storage medium storing a program programmed to perform a dynamic partitioning method according to an embodiment of the present invention comprises: determining whether a consumer without a partition assigned among a plurality of consumers exists; Equally dividing the messages remaining in the first partition in half among the plurality of partitions, and creating two new partitions to store the evenly divided messages in the two new partitions, respectively; Allocating the two new partitions to a consumer without the assigned partition and to a consumer that has been assigned to the first partition; And updating meta data including list information about partitions and mapping information between each consumer and a partition allocated to each consumer. It may be a storage medium that stores a program programmed to perform.

According to the present invention, while maintaining the parallelism, which is the advantage of a fully distributed message system, while maintaining dynamic parallelism, it is possible to solve a load balancing problem due to this heterogeneous workload.

In addition, according to the present invention, it is possible to reduce the overhead due to the redistribution (re-balancing) operation.

1 is a view showing a fully distributed message system.
2 is a flowchart illustrating a message partition redistribution algorithm according to an embodiment of the present invention.
3 is a flowchart illustrating a message partition redistribution algorithm according to another embodiment of the present invention.

1 is a view showing a fully distributed message system.

Referring to FIG. 1, Kafka clusters and consumer groups are shown.

The Kafka cluster 10 may include one or more servers as an element for distributing and processing messages. The Kafka cluster can store messages in multiple servers. The entire messages are divided into a predetermined number of partitions (Partitions: p0, p1, p2, p3), and the partitions can be divided and stored in a plurality of servers (Server 1, Server 2). In this way, a large number of servers can efficiently process large-scale message processing requests by dividing and storing message partitions. Here, the server is a configuration for processing distributed messages, and may be referred to as a broker.

The consumer group 20 may include one or more consumers. The above-described partition is configured to be accessible only through a single consumer belonging to the consumer group to maximize parallelism. That is, in FIG. 1, the consumer (c1) and the consumer (c2) included in the consumer group A are dedicated to the p0, p3 partition, and the p1, p2 partition, respectively, and the consumer group B (consumer group B). Consumers c3, c4, c5, and c6 included in p0, p3, p1, and p2, respectively, may be dedicated. In other words, the p0 and p3 partitions are allocated to the consumer (c1) included in the consumer group A, and the p1 and p2 partitions are allocated to the consumer (c2) included in the consumer group A (consumer group A). Can be.

The message partitioning module (not shown) is an element included in the message system and may perform operations related to metadata management, which will be described later. Specifically, the message partitioning module (not shown) may perform metadata creation and update operations. In addition, the message partitioning module may perform each step of the message partition redistribution algorithm, which will be described later. Meanwhile, the message partitioning module may be a processor in the message system. The message partitioning module may be implemented as a single processor according to an implementation type, and may be implemented as multiple processors, so that each processor may perform some steps of the redistribution algorithm or some operations of each step.

First, for each message partition, the number of processed messages and the number of remaining messages can be managed as metadata.

Here, the metadata may include, for each message partition, at least one of the number of processed messages, the number of remaining messages, and the total number of messages. In one embodiment, the meta data may include offset information. Here, the offset information may be an offset of the total number of messages in each message partition and the number of processed messages, or an offset of the total number of messages in each message partition and the number of remaining messages.

Also, the process of dynamically redistributing partitions to consumers can be invoked periodically.

At this time, in order to maintain the parallelism of the fully-distributed message system, one partition may be processed through only one consumer in one consumer group.

Mapping information for each consumer and message partitions assigned to and processed by each consumer may also be managed as metadata.

Hereinafter, a scheme for dynamically partitioning a partition when a change in the consumer pool occurs based on the fully distributed message system illustrated in FIG. 1 will be described. Here, the consumer pool may be used in the same sense as the above-described consumer group.

Here, there may be three main cases where there is a change in the consumer pool. First, there are cases where new consumers participate in the consumer pool. Second, there may be a case in which at least one consumer among the consumers belonging to the consumer pool has completed processing for the partition to which it is assigned. Third, there may be a case in which one or more consumers among consumers belonging to the consumer pool fail to finish processing for the responsible partition.

2 is a flowchart illustrating a message partition redistribution algorithm according to an embodiment of the present invention.

Before describing the algorithm shown in FIG. 2, a configuration of a distributed message system to which the algorithm is applied will be described. In the distributed message system, there are N partitions and M consumers. Here, the M consumers belong to one consumer group.

The meta data managed by the message partitioning module may include list information and mapping information. Here, the list information may include list information of the N partitions and list information of the M consumers, and the mapping information refers to mapping information between the N partitions and the M consumers. can do.

The list information of the N partitions is P = {p0, p1,... , pi,… , pN}, and the list information of the M consumers is C = {c0, c1, ... , cj,… , cM}. The mapping information may be expressed in a form indicating which consumer is assigned to each partition. Alternatively, the mapping information may be expressed in a form indicating which partition is allocated for each consumer. In one embodiment, the mapping information may be an AllocPatition[N] array. The AllocPatition[N] array may indicate which consumer is assigned an arbitrary partition pi. Here, both P and C mean an active state. That is, an already processed partition or an ended consumer is not managed and can be excluded from the list.

Referring to FIG. 2, first, the message partitioning module determines whether a consumer (cx) having no partition assigned among consumers exists (step 201). Here, the consumer (cx) without the assigned partition may be a consumer newly participating in the corresponding consumer group or a consumer who has completed processing for the partition in charge. If there is no consumer (cx) without an assigned partition, that is, if all consumers are processing a partition in charge of each, the dynamic redistribution algorithm of this cycle ends.

If a consumer without an assigned partition exists, the message partitioning module selects a partition pk in which the messages to be processed remain the most among the partitions P (step 203).

The message partitioning module compares the number of messages remaining in the partition pk with a threshold (step 205). If the number of messages remaining in the partition pk is less than a threshold (in one embodiment, the same case may be included), the dynamic redistribution algorithm of this cycle ends. Here, the threshold may be set based on the number of messages that can make up a sufficient workload even if the remaining messages are divided.

When the number of messages remaining in the partition pk is greater than a threshold (in one embodiment, the same case may be included), the message partitioning module partitions the most remaining messages to be processed (pk) The operation of the consumer (cl) for processing is held, and the messages remaining in the partition (pk) are evenly divided (step 205). In one embodiment, the message partitioning module may divide the messages remaining in the partition pk into two equal partitions pd1 and pd2. Since the messages previously processed in the partition pk are completed messages, the message partitioning module terminates the corresponding partition pk.

The message partitioning module redistributes the newly created message partitions pd1 and pd2 to the consumer cl processing the existing pk and the consumer cx without the assigned partition (step 209).

The message partitioning module updates meta data (list information and mapping information) (step 211).

As such, one period of the dynamic redistribution algorithm is completed, and the dynamic redistribution algorithm of the next period can be repeated. That is, FIG. 2 shows one period of the dynamic redistribution algorithm, and the dynamic redistribution algorithm may be repeated periodically.

According to this dynamic redistribution algorithm, partitions can be distributed to newly available consumers. In particular, since the largest message partition is divided sequentially, it is possible to minimize the shortening of the redistribution period. Meanwhile, since the dynamic redistribution algorithm is performed not periodically but periodically, it is possible to reflect the workload of the consumer that changes every moment.

3 is a flowchart illustrating a message partition redistribution algorithm according to another embodiment of the present invention.

FIG. 3 shows a dynamic redistribution algorithm when one or more of the consumers who processed a message is terminated without completing the processing of the responsible partition.

First, the message partitioning module determines whether any one of the consumers Cf that processed the existing message has failed in an unexpected situation (step 301). In one embodiment, the determination method of step 301 may determine whether the number of messages processed by the consumer for a predetermined time is less than a threshold.

When an event in which any consumer (Cf) of existing consumers processing a message fails due to an unexpected situation, the message partitioning module processes the partition (pa) processed by the terminated consumer (cf). A new partition pb is created using the remaining messages (step 303). That is, the message partitioning module creates a new partition pb composed of the remaining messages. Meanwhile, the existing partition pa may be terminated. In one embodiment, the message partitioning module may terminate the existing partition (pa).

The message partitioning module allocates the created new partition pb to the consumer cx without the assigned partition (step 305).

The consumer failure event described through FIG. 3 is an event that may occur during the message processing operation of the consumers. That is, it may occur during a process in which the period of the dynamic redistribution algorithm described in FIG. 2 is repeated. When the consumer end event occurs, the message partitioning module may create a new partition and wait until step 201. That is, step 305 of FIG. 3 may mean steps 201 and 203 of FIG. 2.

Meanwhile, each of the steps described with reference to FIGS. 2 and 3 may be selectively integrated or divided.

Hereinafter, specific examples according to the dynamic partitioning process described through FIGS. 2 and 3 will be described.

In this example, it is assumed that there are 5 partitions (p0, p1, p2, p3, p4), 5 consumers (c0, c1, c2, c3, c4), and that a total of 100,000 messages are processed. do. At this time, 20,000 messages are distributed to the five partitions p0, p1, p2, p3, and p4, respectively. Each consumer is configured to process one partition. For convenience, it is assumed that c0 processes p0, c1 processes p1, c2 processes p2, c3 processes p3, and c4 processes p4.

In this case, the metadata includes, as list information, partition list information P = {p0, p1, p2, p3, p4} and consumer list information C = {c0, c1, c2, c3, c4}. It may include {c0, p0}, {c1, p1}, {c2, p2}, {c3, p3}, {c4, p4} mapping information AllocPatition[4] (see Table 1 below). Reference)

At this point, at any point in time, the consumer (c4) processed all 20,000 messages assigned to him before other consumers, and at that point, the largest number of messages remained in the partition (p2), and at that point It is assumed that the number of messages remaining in the partition p2 is 10,000. In addition, it is assumed that the threshold is 6,000, and the number of messages remaining in the partition p2 at that time is greater than the threshold.

In this case, the message partitioning module pauses the operation of the consumer (c2) processing the partition (p2), creates two new partitions (p5, p6), and remains in the partition (p2). Divide 10,000 messages into 5000 pieces and allocate them to new partitions (p5, p6), respectively. In addition, the message partitioning module processes the new partitions p5 and p6 to the consumer c2, which processes the existing partition p2, and the consumer c2, which processes all the messages assigned to it. Redistribute.

Meanwhile, the message partitioning module updates meta data. All messages included in the existing partition (p2) have been processed, and since the existing partition (p4) has been divided into new partitions, p2 and p4 are deleted from the partition list, and the new partitions p5 and p6 Is added to the partition list. In addition, the partition mapping information for the consumer c2 that processed the existing partition p2 and the consumer c4 without the assigned partition is updated by processing all the messages assigned to it.

As a result, the updated meta data includes partition list information P = {p0, p1, p3, p5, p6}, and consumer list information C = {c0, c1, c2, c3, c4} as list information. It may include, {c0, p0}, {c1, p1}, {c2, p5}, {c3, p3}, and {c4, p6} mapping information AllocPatition[4].

The table below compares metadata information before and after redistribution.

The above-described example is for a case where a change in the consumer pool occurs due to the completion of processing of a partition in which the consumer is in charge, or when a new consumer participates in the consumer pool or the consumer fails to finish processing for the responsible partition ( It can be similarly applied in case of failure.

As can be seen in the above example, if a partition having a lot of messages to be processed is partitioned, load balancing can be improved, and the redistribution cycle can be shortened. Since message processing is not possible during the redistribution operation, inducing the redistribution period to be long can prevent system performance from being reduced due to the redistribution operation.

On the other hand, in the above example, the number of messages remaining in the partition p2 at a specific time point is 10,000, and the threshold is 6,000. This situation is a case where the number of messages remaining in the partition is greater than a threshold value, and a sufficient workload can be configured by dividing the remaining messages. On the other hand, if the number of remaining messages is not sufficient, partitioning the partition may not have much meaning. Rather, if the number of remaining messages is insufficient, partitioning the partition may degrade system performance due to the redistribution operation. The above-described example is configured to not perform dynamic partitioning when the number of remaining messages does not exceed a predetermined threshold, thereby preventing system performance from deteriorating.

According to the present invention, while maintaining maximization of parallelism, which is the advantage of a fully distributed message system, a load balancing problem due to heterogeneous workload can be solved through a dynamic partitioning technique. In addition, according to the present invention, it is possible to reduce the overhead due to the redistribution (re-balancing) operation.

With the development of the Internet of Things (IoT) in the future, when various types of messages are interlocked through a distributed message system, a load balancing problem due to a static partitioning technique may continuously occur. According to the present invention, by implementing a dynamic partitioning technique with relatively minimal effort, a distributed message system can greatly assist in maintaining high performance message processing performance.

Each step of the message redistribution algorithm according to an embodiment of the present invention described above may be programmed in software, and the programmed program may be stored in a storage medium.

Each component constituting the module, unit or device may be processors that execute successive processes stored in a memory (or storage unit). Each of the steps described in the above-described embodiment can be performed by hardware/processors. Each of the constituent units described in the above-described embodiment can operate as a hardware/processor. Also, the methods proposed by the present invention can be executed as code. This code can be written to a storage medium that can be read by a processor, and thus can be read by a processor provided by an apparatus.

For convenience of explanation, each drawing is divided and described, but it is also possible to design to implement a new embodiment by merging the embodiments described in each drawing. Also, it is within the scope of the present invention to design a computer-readable recording medium in which a program for executing the above-described embodiments is recorded, according to the needs of a person skilled in the art.

The apparatus and method according to the present invention is not limited to the configuration and method of the embodiments described as described above, the above-described embodiments may be a part or all of each of the embodiments selectively so that various modifications can be made It may be configured in combination.

Meanwhile, it is possible to implement the method proposed by the present invention as a code readable by a processor in a recording medium readable by a processor provided in a network device. The processor-readable recording medium includes all kinds of recording devices in which data that can be read by the processor are stored. Examples of the recording medium readable by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the processor readable recording medium may be distributed over network coupled computer systems so that the processor readable code is stored and executed in a distributed fashion.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications can be implemented by a person having ordinary knowledge in the course, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

In addition, in the specification, both the invention of the object and the invention of the method are described, and the description of both inventions may be applied supplementally as necessary.

It is understood by those skilled in the art that various changes and modifications are possible in the present invention without departing from the spirit or scope of the present invention. Accordingly, the present invention is intended to cover modifications and variations of the invention provided within the scope of the appended claims and their equivalents.

In this specification, both the device and method inventions are mentioned, and the descriptions of both the device and method inventions can be applied to complement each other.

10: Kafka cluster
20: Consumer group

Claims (15)

A plurality of partitions for partitioning and storing messages;
A consumer group comprising a plurality of consumers,
Messages stored in one partition are processed by one consumer included in the consumer group; And
While the consumer is processing the partition,
A message partitioning module that manages metadata including list information for the plurality of partitions and mapping information between each consumer and a partition assigned to each consumer; Including,
The message partitioning module updates the metadata each time a dynamic redistribution operation is performed,
The dynamic redistribution operation is performed by the message partitioning module:
A process of determining whether a consumer without an assigned partition exists among the plurality of consumers;
A first partition having a maximum number of messages is selected from a plurality of partitions,
A process of equally dividing the messages remaining in the first partition in half, creating two new partitions, and storing the evenly divided messages in the two new partitions, respectively; And
A process of allocating the two new partitions to a consumer without the assigned partition and a consumer that has been assigned to the first partition; Containing,
Distributed message system.

delete According to claim 1,
The dynamic redistribution operation is performed according to a predetermined period or whenever a change occurs in the consumer group.
The method of claim 3,
When a change occurs in the consumer group,
When a new consumer joins the consumer group, when at least one consumer among the plurality of consumers processes all messages stored in a partition assigned to the consumer, and at least one consumer among the plurality of consumers Distributed message system, characterized in that any one of the cases that terminated without processing the message stored in the assigned partition.
delete According to claim 1,
And a process for generating the metadata before the determining process.
According to claim 1,
The first partition is a distributed message system, characterized in that the message is the most remaining message to be processed among the plurality of partitions.
According to claim 1,
A process of comparing the number of messages remaining in the first partition with a threshold value after the determining process; Distributed message system further comprising a.
According to claim 1,
The meta data further comprises a list information for the plurality of consumers distributed message system.
In the dynamic partitioning method, messages stored in one partition are processed by one consumer included in a consumer group including a plurality of consumers,
Storing a plurality of partitions by dividing the messages;
Managing meta data including list information for the plurality of partitions and mapping information between each consumer and a partition allocated to each consumer;
Updating the metadata each time a dynamic redistribution operation is performed while the one consumer processes the one partition;
Determining whether there is a consumer without a partition assigned among the plurality of consumers; Including,
A first partition having a maximum number of messages is selected from a plurality of partitions,
The messages remaining in the first partition are equally divided in half, and two new partitions are created to store the evenly divided messages in the two new partitions,
Assigning the two new partitions to a consumer without the assigned partition and to a consumer that has been assigned to the first partition,
Dynamic partitioning method in distributed message system.

delete The method of claim 10,
The meta data, dynamic partitioning method in a distributed message system further comprises a list information for the plurality of consumers.
The method of claim 10,
The first partition is a dynamic partitioning method in a distributed message system, characterized in that the message is the most remaining partition to be processed among the plurality of partitions.
The method of claim 10,
After the determining step, comparing the number of messages remaining in the first partition with a threshold value; Dynamic partitioning method in a distributed message system further comprising a.
Storing a plurality of partitions by dividing the messages;
Managing meta data including list information for the plurality of partitions and mapping information between each consumer and a partition allocated to each consumer;
Updating the metadata whenever a dynamic redistribution operation is performed while the consumer is assigned to the partition for the consumer;
Selecting a first partition having a maximum number of messages from a plurality of partitions;
Equally dividing the messages remaining in the first partition in half, creating two new partitions, and storing the equally divided messages in the two new partitions;
Allocating the two new partitions to a consumer without the assigned partition and to a consumer that has been assigned to the first partition; A computer-readable storage medium storing a program programmed to perform the operation.

Images