WO2022037293A1

WO2022037293A1 - Message-oriented middleware layout method and apparatus, server, and storage medium

Info

Publication number: WO2022037293A1
Application number: PCT/CN2021/104775
Authority: WO
Inventors: 杨振
Original assignee: 中兴通讯股份有限公司
Priority date: 2020-08-18
Filing date: 2021-07-06
Publication date: 2022-02-24
Also published as: CN114153623A

Abstract

The embodiments of the present application relate to the technical fields of messages and communication, and disclose a message-oriented middleware layout method and apparatus, a server, and a storage medium. The method comprises: acquiring topic coupling sets, the topics in each coupling set having a coupling relationship; on the basis of the topic coupling sets, performing decoupling and grouping of all of the topics to obtain a plurality of topic sets; and respectively deploying the plurality of topic sets in message-oriented middleware container clusters, one topic set being deployed in one message-oriented middleware container cluster.

Description

Layout method, device, server and storage medium of message middleware

cross reference

This application is based on the Chinese patent application with the application number "202010833757.4" and the application date is August 18, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference. Apply.

technical field

The embodiments of the present application relate to the field of information and communication technologies, and in particular, to a layout method, device, server, and storage medium for message middleware.

Background technique

Taking the business scenario of message middleware Kafka passing messages between multiple services as an example, there are often the following problems in the process of using message middleware: (1) The first scenario is shown in Figure 1, where there is Service A (Service A ) and Service B (Service B), in actual business, A is both a producer of B and a consumer of B; B is both a producer of A and a consumer of B; A and B share the same Kafka service , which will cause the coupling of (A, B). (2) The second scenario is shown in Figure 2. There are Service A, Service B, and Service C. B is a consumer of A and a producer of C. When two sets of services access the same Kafka service, they will be upgraded or When some Kafka failure problems occur, there will be situations where both sets of services cannot use the Kafka service. That is, the coupling between (A, B) and (B, C). (3) The third scenario is shown in Figure 3. There are Service A, Service B, Service C and Service D. Due to the different importance of the business data of (A, B) and (C, D), the number of copies of Topic1 and Topic2 It is also different; the number of partition replicas (Replication) of each topic is positively related to the importance of business data. The more important the business data is, the more partition replicas there are. According to the characteristics of Kafka itself, the number of partition replicas is the number of copies. Too many partition replicas can The throughput increases, and the write pressure also increases. In addition, cumbersome environment configuration and deployment operations also bring technical difficulties to developers.

SUMMARY OF THE INVENTION

An embodiment of the present application provides a layout method for message middleware, including: acquiring a coupling set of topics; wherein, topics in each coupling set have a coupling relationship; decoupling and grouping all topics according to the coupling set of topics, to obtain Multiple Topic sets; multiple Topic sets are deployed in each message middleware container cluster, where one Topic set is deployed in one message middleware container cluster.

An embodiment of the present application further provides a layout device for message middleware, including: an acquisition module, used for acquiring a coupling set of topics; wherein, topics in each coupling set have a coupling relationship; and a grouping module is used for coupling according to topics The set decouples and groups all topics to obtain multiple topic sets; the cluster deployment module is used to deploy multiple topic sets in each message middleware container cluster, where one topic set is deployed in one message middleware container in the cluster.

An embodiment of the present application further provides a server, including: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are processed by the at least one processor The processor executes, so that at least one processor can execute the layout method of the message middleware as described above.

Embodiments of the present application further provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the above-mentioned layout method of the message middleware.

Description of drawings

FIG. 1 is a schematic diagram of a path for message delivery by message middleware in a first scenario in the background technology of the present application;

FIG. 2 is a schematic diagram of a path for message delivery by message middleware in a second scenario in the background technology of the present application;

FIG. 3 is a schematic diagram of a path for message delivery by message middleware in a third scenario in the background technology of the present application;

4 is a flowchart of a layout method for message middleware according to the first embodiment of the present application;

5 is a flowchart of a layout method for message middleware according to a second embodiment of the present application;

6 is a flowchart of decoupling and grouping all Topics by adopting a greedy algorithm in the layout method of message middleware according to the second embodiment of the present application;

7 is a schematic structural diagram of a layout device for message middleware according to a third embodiment of the present application;

8 is a schematic structural diagram of a container multi-cluster distributed system according to the message middleware in the fourth embodiment of the present application;

FIG. 9 is a schematic structural diagram of a server according to a fifth embodiment of the present application.

detailed description

In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clear, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that, in each embodiment of the present application, many technical details are provided for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized. The following divisions of the various embodiments are for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the various embodiments may be combined with each other and referred to each other on the premise of not contradicting each other.

Message middleware, also called message queue middleware, is an intermediate component that uses an efficient and reliable message delivery mechanism for platform-independent data exchange. It includes a message server (Broker), a producer (Producer), a consumer (Consumer), and consumption topics. (Topic) et al. Among them, the Broker provides the core message service for the server, the Producer is responsible for producing messages, the Consumer is responsible for consuming and processing messages, and the Topic is used to collect messages in the publish-subscribe mode. In the scenario of a multi-service system, using message middleware to deliver messages can effectively solve the problem of business coupling. In the process of interacting with the message middleware, the message middleware service is used in multiple services or in the application scenario of the service architecture. Generally, the form of a shared single node or multi-node cluster is used to realize the consumption and production of messages, and as the importance of service data increases, Increase, the number of message backups increases, the throughput for writing message middleware increases, and the corresponding IO pressure also increases.

However, in the process of using message middleware, there may be the interaction of production and consumption between services, but with the increase of business requirements, the number of services also increases, and there is an inevitable coupling between services, which leads to message middleware. The working stability of the software cannot meet the needs of users. Taking the business scenario of message middleware Kafka passing messages between multiple services as an example, there are often the following problems in the process of using message middleware: (1) The first scenario is shown in Figure 1, where there is Service A (Service A ) and Service B (Service B), in actual business, A is both a producer of B and a consumer of B; B is both a producer of A and a consumer of B; A and B share the same Kafka service , which will cause the coupling of (A, B). (2) The second scenario is shown in Figure 2. There are Service A, Service B, and Service C. B is a consumer of A and a producer of C. When two sets of services access the same Kafka service, they will be upgraded or When some Kafka failure problems occur, there will be situations where both sets of services cannot use the Kafka service. That is, the coupling between (A, B) and (B, C). (3) The third scenario is shown in Figure 3. There are Service A, Service B, Service C and Service D. Due to the different importance of the business data of (A, B) and (C, D), the number of copies of Topic1 and Topic2 It is also different; the number of partition replicas (Replication) of each topic is positively related to the importance of business data. The more important the business data is, the more partition replicas there are. According to the characteristics of Kafka itself, the number of partition replicas is the number of copies. Too many partition replicas can The throughput increases, and the write pressure also increases. In addition, cumbersome environment configuration and deployment operations also bring technical difficulties to developers.

The main purpose of the embodiments of the present application is to propose a layout method of message middleware, so that the service pressure of multi-service access message middleware is dispersed, the coupling between multiple services accessing message middleware is reduced, and the message middleware is improved. work stability.

The first embodiment of the present application relates to a layout method for message middleware, including: acquiring metadata of each consumption topic topic; generating a coupled set of topics according to the metadata of the topic; wherein, the topics in each coupled set have a coupling relationship ; Decouple and group all Topics according to the Topic coupling set to obtain multiple Topic sets; Deploy multiple Topic sets in each message middleware container cluster, where one Topic set is deployed in a message middleware container cluster middle. In this embodiment, the layout method of the message middleware is applied to a multi-cluster distributed system, the execution subject is the server serving as the message middleware container bearing node in the system, and the message middleware uses Kafka, a high-throughput distribution system. Take the publish-subscribe messaging system as an example to illustrate.

The present embodiment will be further elaborated below with reference to the accompanying drawings. The layout method of the message middleware in this embodiment is shown in FIG. 4 , and specifically includes:

In step 401, metadata of each consumption topic Topic is acquired.

Specifically, the metadata of the topic is used to describe the attribute information of the topic. The attributes described by the metadata of the topic include: the capacity index of the topic, that is, the indicator of the importance of business data; the set of producers under the topic, that is, the The service collection of the producer of the topic; the service collection of the consumers under the topic. The Topic's capacity indicator refers to the number of Topic message backups or copies.

In Kafka, the capacity of a topic is generally described by the number of replica copies. The topic capacity indicator is also an indicator of the importance of business data. In order to ensure the disaster recovery capability of business data, the more important business data is, the larger the number of replica copies. In addition, in other types of message middleware, the form of pre-set importance can also be used, and the importance of each topic can be scored by the operation and maintenance personnel. At this time, the importance of the topic is also determined by the attribute information. data is described.

Step 402, generating a coupled set of topics according to the metadata of the topics.

Specifically, the coupling set is a set that describes the coupling relationship between services in the service topic metadata. In order to obtain the specific coupling relationship between all topics, it is necessary to establish a coupling set of multiple topics first. Among them, in each Topic coupling set, all Topics have a coupling relationship.

Further, in an example, the way to establish a coupled set is usually to describe the relationship between services using topics, and then to find the intersection of the producer set and the consumer set under all topic metadata, if the intersection does not If it is empty, it is considered that there is coupling between the services of the two topics, and the coupling set is added, that is, the topics with the same service in the producer service set or the consumer service set are generated as a coupled set. After traversing the metadata of all Topics once, a complete Topic coupling set is generated, and then the operation is continued from the remaining Topics until all Topics are put into the Topic coupling set.

In an example, service metadata can also be used to describe service attributes, and the attributes described by the service metadata include: basic service information, a Topic collection of the service as a producer, and a Topic collection of the service as a consumer. It can be found that both the service metadata and the topic metadata can describe the relationship between the service and the message middleware, and the difference lies in the different description directions. Therefore, in this embodiment, a coupled set of topics may also be generated according to service metadata, or a coupled set of topics may be generated according to service metadata and topic metadata. It is worth noting that in this embodiment, whether service metadata or topic metadata is used to generate the coupling set of topics, the coupling relationship between services is described with the topic as the center.

In one example, topic metadata and service metadata may be represented by service message tuples and topic message tuples, respectively.

Among them, Topic message tuple Ti=(Rc _i , PrSet _i , CoSet _i ), in which, TC (Topic Capacity) represents the capacity of the topic; ProSet (Producer Set) represents the set of producers under the topic, and the set contains all the producers of the topic. Topic services; ConSet (Consumer Set) represents the set of consumers under the Topic, and the set contains all the services that consume the Topic.

The service message tuple S={Info, ProTopic, ConTopic}, where Info represents basic service information, ProTopic represents the Topic set as the producer, and ConTopic represents the Topic set as the consumer.

In an example, the process of generating the Topic coupling set by using the service message tuple and the Topic message tuple includes: establishing the association relationship between the service and the Topic according to the Topic message tuple and the service message tuple Relation=(T _i , PS _j , CS _k ), generate an association relationship set {(T _i , PS _j , CS _k )}, according to the definition of the coupling relationship, PS _j and CS _j belong to the same service, so the association relationship can be intersected to generate a coupling set {(T _i , T _j )}.

Step 403 , decoupling and grouping all the Topics according to the Topic coupling set to obtain multiple Topic sets.

Specifically, in order to reduce the coupling between services after deployment, topics need to be decoupled and grouped. The main way is to put all topics with coupling relationship in a set, as a topic group, and deploy the topic group in a cluster mode during actual deployment, so as to eliminate the coupling relationship between topic clusters and improve the The stability of the operation of the message middleware in the distributed system. At the same time, through this grouping method, the number of topic sets after grouping can be reduced as much as possible, and the efficiency of message middleware deployment can be improved.

Step 404: Deploy multiple topic sets in each message middleware container cluster respectively. Among them, a topic set is deployed in a cluster of message middleware containers.

Specifically, obtaining multiple topic sets through decoupling grouping can be understood as obtaining the specific layout of deploying these topics through decoupling grouping. In addition, each Topic set can deploy different numbers of message middleware, thus forming message middleware clusters of different scales. The number of containers in the message middleware cluster can be deployed according to the number of topics or the actual expected load of the topic cluster.

In an example, according to the characteristic that the larger the topic capacity is, the higher the degree of importance is, in order to ensure the disaster recovery capability of business data and the reliability of the message middleware service, the average capacity of the topic collection can be used as an indicator to measure the entire topic collection. Based on the importance level, the number of containers required to deploy the message middleware container cluster of the topic set is determined according to the importance level of the topic set. The average capacity is the ratio of the topic set capacity to the number of topics. A cluster with a larger average capacity indicates that the message middleware messages on the cluster are more important, and more containers need to be allocated, that is, the larger the cluster size is. Ultimately, the number of message backups in the cluster is the product of the number of middleware containers and the number of topic copies. A large number of data backups ensure the reliability of important topics.

In practical applications, it is assumed that Topic set A={T ₁ , T ₂ , T ₃ , T ₄ , T ₅ }, Topic set B={T ₆ , T ₇ , T ₈ , T ₉ }. Thus, there are message middleware clusters S _A and S _B corresponding to Set A and Set B. Assuming that the number of T ₁ , T ₂ , T ₃ , T ₄ , and _T ₅ replicas are 9, 2, 3, 7, and 9, respectively, the average capacity of cluster SA

The number of copies of T ₆ , T ₇ , T ₈ , and T ₉ are 4, 1, 8, and 3, respectively, then the average capacity of cluster _SB

The ratio of the preset average capacity to the number of message middleware containers is 2:1, then finally three containers are allocated to the message middleware cluster _{SA and two containers are allocated to S B} _, so as to achieve uniform load distribution among the containers , to reduce the read and write pressure of containers that carry more important topics.

It should be noted that the above examples in this embodiment are all examples for easy understanding, and do not limit the technical solutions of the present application.

Compared with the related art, the layout method of the message middleware in this embodiment decouples and groups all the topics, deploys the message middleware in a cluster formed by the message middleware cluster, and distributes the topics with the coupling relationship. Layout into a message middleware container cluster, thereby reducing the coupling between the message middleware container clusters deploying each topic, dispersing the service pressure of multi-service access to message middleware, thereby improving the overall reliability of the message middleware cluster , to avoid the possibility of large-scale service paralysis caused by single-container cluster downtime. Moreover, since each topic deployed in a message middleware cluster is a topic without a coupling relationship, the coupling between multiple services accessing the message middleware can be reduced, thereby improving the working stability of the message middleware, and at the same time. It can simplify the deployment process of message middleware. In addition, for topic sets with high average capacity, that is, container clusters with high importance, more containers are allocated to further ensure the reliability of important services.

The second embodiment of the present application relates to a layout method for message middleware, which is roughly the same as the first embodiment of the present application. The main difference is that in this embodiment, all topics are decoupled according to the coupling set of topics. Grouping to obtain multiple topic sets, including: using an improved greedy algorithm to decouple and group topics. In this embodiment, the selection condition of the improved greedy algorithm is that the topic currently traversed does not belong to the same coupling set as the topic that has been selected in this traversal. The end conditions include: the total capacity of the topics selected in this traversal is greater than or equal to the preset capacity and the initial topic set is empty.

The present embodiment will be further described below with reference to the accompanying drawings. The layout method of the message middleware in this embodiment is shown in FIG. 5 , and specifically includes:

Step 501: Obtain metadata of each consumption topic topic.

Step 502 , generating a coupled set of topics according to the metadata of the topics.

Steps

501 and 502 are substantially the same as

steps

401 and 402 in the first embodiment of the present application, and the relevant implementation details have been specifically described in the first embodiment of the present application, and are not repeated in this embodiment.

Step 503 , using an improved greedy algorithm, decoupling and grouping all Topics according to the Topic coupling set to obtain multiple Topic sets.

Specifically, the greedy algorithm in this embodiment is characterized in that the traversal objects of the greedy algorithm are preprocessed, that is, an ordered set of topics is generated after sorting all topics according to their importance, that is, in this embodiment For the mentioned initial set, the greedy strategy of the greedy algorithm will perform the traversal process in the order of the elements in the set. In addition, the selection conditions in the greedy strategy are: the topic currently traversed does not belong to the same coupling set as the topic selected in this traversal; the end condition is that the total capacity of the topics selected in this traversal is greater than or equal to the preset The capacity as well as the initial set are empty.

In one example, the greedy algorithm is used to decouple and group all Topics according to their coupled sets, and the process of obtaining multiple Topic sets is shown in Figure 6, which specifically includes:

Step 5031: Sort all Topics according to their importance to obtain an initial set.

Specifically, taking Kafka as the message middleware as an example, the importance of a topic is positively related to the number of copies of the topic backup, that is, the importance of a topic is represented by the number of copies of the topic, and the topics in the initial set are ordered according to their importance. arrangement. Subsequent greedy algorithms will access the topics in the initial set in the existing order.

Step 5032, create a topic collection.

Specifically, the newly created Topic set is an empty set. After accessing a Topic that satisfies the selection condition according to the subsequent greedy algorithm, the Topic is selected and added to the Topic set.

Step 5033, traverse the initial set, and access the Topics in the initial set in turn.

Step 5034, determine whether the currently accessed Topic and the selected Topic in the Topic set belong to the same coupled set; if the currently accessed Topic and the selected Topic in the Topic set belong to the same coupled set, then continue to step 5033, traverse the initial Collection; if the currently accessed topic and the selected topic in the topic collection do not belong to the same coupling collection, step 5035 is executed.

Step 5035: Select the currently accessed topic to add to the topic set.

Specifically, each time a Topic is accessed, the Topic that has a coupling relationship with the currently accessed Topic will be searched from all the selected Topics in the newly created Topic set according to the coupling set to which the Topic belongs. If there is a Topic with a coupling relationship, the currently accessed Topic will not be added to the Topic collection, and continue to access the next Topic in the initial collection in order. If the topic accessed this time does not have a coupling relationship with any topic in the newly created topic collection, the currently accessed topic is added to the newly created topic collection.

Step 5036, determine whether the traversal ends; if the traversal ends, execute step 5037; if the traversal does not end, continue to execute step 5033 to traverse the initial set.

Specifically, after the traversal has accessed all topics in the initial collection, the topic collection is saved, that is, a topic collection has been obtained through this traversal.

In an example, you can set the upper limit of the capacity of a topic collection before creating a new topic collection, and before the traversal ends, determine whether the capacity of the topic collection is greater than or equal to the upper limit of the collection capacity after the current topic is added to the topic collection. If the capacity of the Topic collection is greater than or equal to the upper limit of the capacity of the collection, the traversal is ended and the current Topic collection is directly saved. Through this technical means, the capacity of a single Topic set can be controlled, so as to reduce the number of Topic sets as much as possible, control the number of containers required to deploy a message middleware container cluster of a single Topic, so as to avoid the important Topic in a cluster. More, reducing the probability of large-scale service failure of important topics.

Step 5037, update the initial set, and determine whether the updated initial set is empty; if the updated initial set is empty, then execute step 5038; if the updated initial set is not empty, then execute step 5032 again to create a new empty set. Topic collection.

Step 5038, output all topic sets.

Specifically, the method of updating the initial set is: taking the difference set of the Topic included in the initial set and all the Topics of the acquired Topic set as the updated initial set. When the updated initial set is empty, it means that all topics have been decoupled and grouped. At this time, the greedy algorithm is terminated and no longer executed, and all existing topic sets are output as the deployment message middleware container cluster. layout basis.

Moreover, since the Topics in the initial set are sorted according to their importance, the importance of the Topics in the Topics set obtained first will be higher than that of the Topic sets obtained later. In such a greedy algorithm, the execution efficiency is high, and the Topics with higher importance will be preferentially assigned to the same Topic set, and the number of the finally generated Topic sets will be reduced. At the same time, when deploying message middleware containers corresponding to important topic sets, allocating more containers can simultaneously improve the reliability of these highly important topics, thus simplifying the deployment and deployment of message middleware clusters by operation and maintenance personnel. maintenance strategy.

Step 504: Deploy multiple topic sets in each message middleware container cluster respectively.

Step 504 is substantially the same as step 404 in the first embodiment of the present application, and the relevant implementation details have been specifically described in the first embodiment of the present application, and will not be repeated in this embodiment.

The steps of the above various methods are divided only for the purpose of describing clearly. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this patent. ;Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

Compared with the related art, in the layout method of the message middleware in this embodiment, only when all topics are decoupled and grouped, an improved greedy algorithm can be used to sort the topics in the initial set traversed by the greedy algorithm, It makes the topics with high importance priority to be grouped into the same topic collection. When deploying the corresponding message middleware container for the topic collection, more containers are allocated, which disperses the pressure of important topics to access service messages, and further Improves the reliability of important message middleware. At the same time, it can simplify the strategy for deploying and maintaining message middleware clusters for operation and maintenance personnel.

The third embodiment of the present application relates to a layout device for message middleware, as shown in FIG. 7 , including:

The obtaining module 701 is configured to obtain a coupling set of Topics; wherein, the Topics in each coupling set have a coupling relationship.

In one example, the generating module 701 may also be configured to generate the Topic coupling set according to the service message tuple, or simultaneously according to the service message tuple and the Topic message tuple.

In another example, the generating module 701 can also be used to generate the Topic with the same service in the producer service set or the consumer service set as a coupled set.

The grouping module 702 is used for decoupling and grouping all the Topics according to the coupling set of Topics to obtain multiple Topic sets.

In an example, the grouping module 702 can also be used to decouple the Topic by adopting an improved greedy algorithm. The specific process of decoupling grouping includes: sorting all Topics according to their importance to obtain the initial set; traversing the initial set using a greedy strategy, and placing the selected Topics in the same Topic set; among them, the selection conditions of the greedy strategy is: the currently traversed Topic does not belong to the same coupled set as the Topic selected in this traversal; the difference between the Topic contained in the initial set and all the Topics in the acquired Topic set is used as the updated initial set; yes The updated initial set repeats the traversal of the greedy strategy to obtain a new topic set and the update of the initial set, until the updated initial set is empty.

The cluster deployment module 703 is configured to deploy multiple topic sets in each message middleware container cluster, wherein one topic set is deployed in one message middleware container cluster.

In an example, the cluster deployment module 703 may also be configured to determine the number N of message middleware containers required by the deployed message middleware container cluster according to the number of topics in the topic set and the number of copies of each topic; deploy the topic set in N message middleware containers.

In another example, the cluster deployment module 704 can also be configured to calculate the average capacity of the Topic set according to the number of Topics and the number of copies of each Topic backup; N is determined according to the average capacity of the Topic set, where N and the average of the Topic set capacity is positively correlated.

It is not difficult to find that this embodiment is a virtual device embodiment corresponding to the first embodiment or the second embodiment, and this embodiment can be implemented in cooperation with the first embodiment or the second embodiment. The related technical details mentioned in the first embodiment or the second embodiment are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the related technical details mentioned in this embodiment can also be applied to the first embodiment or the second embodiment.

It is worth mentioning that all modules involved in this implementation are logical modules. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. combination implementation. In addition, in order to highlight the innovative part of the present application, this embodiment does not introduce units that are not closely related to solving the technical problem raised by the present application, but this does not mean that there are no other units in this embodiment.

The fourth embodiment of the present application relates to a container multi-cluster distributed system of message middleware, and this embodiment may include a layout device for message middleware described in the third embodiment of the present application.

This embodiment will be further elaborated below with reference to the accompanying drawings. The container multi-cluster distributed system of the message middleware in this embodiment is shown in FIG. 8 , including:

The basic environment module 801 is used to automatically build and maintain the automatically compiled running environment, construction tool, installation environment, and image running environment through one-click scripting.

Specifically, the construction of the basic environment includes not only the construction of the compilation environment, but also the construction and installation of the container environment. This module mainly builds the compilation environment according to the requirements of the Kafka source code, including the detection and installation of the Java environment and the adaptation and installation of the Scala environment. For the adaptation installation of the Gradle environment, since the Kafka source code will have a recommended environment version for installation, it is necessary to adapt the environment installation package according to the Gradle configuration file to build a compilation environment; in addition, a container environment needs to be built, which is mainly docker and docker- compose environment.

The source code compilation module 802 is used for compiling and packaging the customized open source message middleware source code into a compressed package form through a series of standardized operations.

Specifically, on the basis of the completion of the basic environment module, the source code compilation module imports the customized Kafka source code developed and tested by the user, uses a one-click compilation script, starts a series of standardized compilation actions, and uses the Gradle compilation tool to package Kafka into the form of an installation package.

Among them, the personalized customized message middleware source code refers to the compilable source code that the user performs secondary development according to his own business scenario requirements or the open source message middleware source code with recompilation requirements and passes all tests. A series of table conversion operations refers to encapsulating the construction of the basic environment and compilation environment construction tools through an automated script. The automated script includes the installation of the basic environment according to the source code requirements, and the update of the source code of the message middleware and the pull of the compilation dependencies. Take and install the package.

The image generation module 803 is configured to generate a message middleware image supporting the system, a basic image required by the system, and an upper-layer application image supporting the message middleware.

Specifically, to package the container image required for topic collection and collection monitoring, the installation package used by the Kafka image packaged by this module comes from the installation package generated by the source code compilation module, and the installation packages of the other two images Zookeeper and Kafka-Eagle are required. Pull from the official website; build the image mainly through Shell script and Dockerfile file, and update the image to the image warehouse for production environment use.

The service management module 804 is configured to rearrange and allocate services to different message middleware clusters from two dimensions of the importance of business data of multiple services and the mutual coupling relationship of services.

Specifically, the service management module mainly realizes the rearrangement of the Kafka cluster assigned to each microservice. The module first abstracts the microservice information data from the topic microservice metadata, and can extract the microservices by producing the topic collection and consuming the topic collection. The coupling relationship between the microservices and the topic is rearranged in the cluster by adopting an improved greedy algorithm centered on the topic.

The cluster deployment module 805 is configured to deploy the laid out Topics on the message middleware multi-cluster through the container cluster metadata to implement the container cluster group.

Further, the cluster deployment module is also used to construct a microservice layout cluster from the image generated by the image generation module through the cluster management tool (docker-compose, docker-manage) to form a container cluster group, and the cluster group is composed of a plurality of container cluster elements. According to the concept of container cluster meta, each container cluster corresponds to a topic set.

The security authentication module 806 is configured to realize the security authentication access of the client through dynamic SSL certificate update.

Specifically, the security authentication module in this embodiment is mainly used to manage the dynamic update of the SSL authentication of the user client, the ACL authorization control of Zookeeper, and the security authentication required by Kafka itself.

The unified interface module 807 is configured to provide interface access for external clients through the packaged interface configuration.

Specifically, the unified interface module in this embodiment provides a unified interface between the client and the server for the cloud management platform. The server interface is mainly used in the actual deployment environment. Topic is created in each cluster environment, and the interface includes the interface created by the topic. A series of parameters can be created by user customization; the client interface includes the configuration parameter interface for the producer and consumer clients to connect to the Kafka server.

The operation monitoring module 808 is used for implementing message middleware cluster monitoring by using an existing open source message middleware monitoring tool.

Specifically, the operation monitoring module mainly uses the Kafka monitoring container built by the image-packaged Kafka-Eagle image to monitor the Kafka topic status, the throughput of production and consumption messages, the status of the container cluster, and event alarms.

The fifth embodiment of the present application relates to a server, as shown in FIG. 9 , comprising: at least one processor 901; and a memory 902 connected in communication with the at least one processor 901; Instructions executed by the processor 901, the instructions are executed by at least one processor 901, so that the at least one processor 901 can execute the layout method of the message middleware in the first or second embodiment.

The memory 902 and the processor 901 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 901 and various circuits of the memory 902 together. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor 901 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 901 . Processor 901 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 902 may be used to store data used by the processor 901 when performing operations.

The sixth embodiment of the present application relates to a computer-readable storage medium storing a computer program. The above method embodiments are implemented when the computer program is executed by the processor.

That is, those skilled in the art can understand that all or part of the steps in the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for realizing the present application, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present application. Scope.

Claims

A layout method for message middleware, applied to a multi-cluster distributed system, the method comprising:

Obtain a coupled set of topics; wherein, the topics in each of the coupled sets have a coupling relationship;

All the Topics are decoupled and grouped according to the Topic coupling set to obtain multiple Topic sets;

A plurality of the topic sets are respectively deployed in each message middleware container cluster, wherein one of the topic sets is deployed in one message middleware container cluster.
The layout method of the message middleware according to claim 1, wherein the decoupling grouping of all the Topics according to the Topic coupling set is performed to obtain multiple Topic sets, including:

Sort all the Topics according to the importance of the Topics to obtain an initial set;

The initial set is traversed by a greedy strategy, and the selected Topics are placed in the same Topic set; wherein, the selection condition of the greedy strategy is: the currently traversed Topic and the selected Topic in this traversal do not belong to the same a said coupled set;

Use the difference set of the Topic contained in the initial set and all Topics of the acquired Topic set as the updated initial set;

The traversal of the greedy strategy is repeatedly performed on the updated initial set to obtain a new topic set and an update of the initial set, until the updated initial set is empty.
The layout method of message middleware according to claim 2, wherein the traversing the initial set using a greedy strategy comprises:

If the total capacity of the topics selected in this traversal is greater than or equal to the preset capacity, the traversal ends.
The layout method of message middleware according to claim 2 or 3, wherein the importance of the topic is positively related to the number of copies of the topic backup.
The layout method for message middleware according to any one of claims 1 to 4, wherein before the acquiring the coupling set of topics, further comprising:

Determine the set of producer services and the set of consumer services of the Topic according to the metadata of the Topic;

Generating Topics with the same service in the producer service set or the consumer service set as a coupled set.
The method for laying out message middleware according to any one of claims 1 to 5, wherein the deploying a plurality of the topic sets respectively in each message middleware container cluster comprises:

Determine the number N of message middleware containers required by the deployed message middleware container cluster according to the number of topics in the topic set and the number of copies of each topic;

The Topic set is deployed in N message middleware containers.
The method for laying out message middleware according to claim 6, wherein, according to the number of topics in the topic set and the number of copies of each topic, the number of message middleware containers required by the deployed message middleware container cluster is determined. Quantity N, including:

Calculate the average capacity of the Topic set according to the number of Topic and the number of copies of each Topic backup;

The N is determined according to the average capacity of the Topic set, where the N is positively correlated with the average capacity of the Topic set.
A layout device for message middleware, comprising:

an acquisition module for acquiring a coupled set of topics; wherein, the topics in each of the coupled sets have a coupling relationship;

a grouping module, configured to decouple and group all the Topics according to the coupling set of the Topics to obtain multiple Topic sets;

The cluster deployment module is configured to deploy a plurality of the topic sets in each message middleware container cluster, wherein one of the topic sets is deployed in one message middleware container cluster.
A server that includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any one of claims 1 to 7 The layout method of the message middleware described above.
A computer-readable storage medium storing a computer program, when the computer program is executed by a processor, the method for laying out the message middleware according to any one of claims 1 to 7 is implemented.