CN117170969A

CN117170969A - Operation and maintenance method, partition equilibrium recovery device, equipment and medium of server cluster

Info

Publication number: CN117170969A
Application number: CN202311099050.5A
Authority: CN
Inventors: 孟江; 冯子杰; 杨旭杰; 梁海燕
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-08-29
Filing date: 2023-08-29
Publication date: 2023-12-05

Abstract

The disclosure provides an operation and maintenance method, a partition equilibrium recovery device, equipment and a medium for a server cluster, which can be applied to the technical fields of financial science and technology and operation and maintenance. The operation and maintenance method comprises the following steps: acquiring message attribute information uploaded to a distributed server cluster by a production end, wherein the message attribute information comprises m message topic information and n message partition information corresponding to the message topic information; calculating the message data amount in each message partition information aiming at each message topic information; determining a data quantity difference value according to the message data quantity in the message partition information; and under the condition that the data quantity difference value meets a preset difference threshold value, adjusting a transmission mode of the production end, wherein the production end uploads the message data into the message partition information through the transmission mode.

Description

Operation and maintenance method, partition equilibrium recovery device, equipment and medium of server cluster

Technical Field

The present disclosure relates to the field of financial science and technology and operation and maintenance technology, and in particular, to an operation and maintenance method, a partition balancing recovery device, an electronic device, a computer readable storage medium, and a computer program product for a distributed server cluster.

Background

In the practical use of the distributed server cluster, in order to effectively promote the concurrency of the system, the message theme of the distributed server cluster is set into multiple partitions. But often a situation where the partition data is unbalanced. When a plurality of partitions of a certain message subject only have a small amount of data for a long time, and the partitions have no data, the operation safety of the distributed server cluster may be affected.

The current abnormal state of unbalanced partition data mainly comes from passive waiting for the abnormal report found by the client application, and no active detection optimization scheme exists. In the face of the requirement of message processing capacity of several trillion magnitude, the existing abnormal discovery mode is low in timeliness and high in labor cost, and the operation safety of the distributed server cluster can be reduced.

Disclosure of Invention

In view of the foregoing, the present disclosure provides an operation and maintenance method, a partition balancing restoration apparatus, an electronic device, a computer readable storage medium, and a computer program product of a distributed server cluster that improve the operation security of the distributed server cluster.

According to a first aspect of the present disclosure, there is provided an operation and maintenance method of a distributed server cluster, applied to a partition balancing recovery device, the method including:

Acquiring message attribute information uploaded to the distributed server cluster by a production end, wherein the message attribute information comprises m message topic information and n message partition information corresponding to the message topic information, m is an integer greater than 0, and n is an integer greater than 1;

calculating the message data amount in each message partition information according to each message topic information;

determining a data quantity difference value according to the message data quantity in the message partition information;

and under the condition that the data quantity difference value meets a preset difference threshold value, adjusting a transmission mode of the production end, wherein the production end uploads message data into the message partition information through the transmission mode.

According to an embodiment of the present disclosure, the operation and maintenance method further includes:

acquiring cluster attribute information of the distributed server cluster under the condition that the data volume difference value does not meet a preset difference threshold, wherein the cluster attribute information comprises the unconsumed data volume of message data in the message partition information by different consuming terminals;

determining an unconsumed difference value according to the unconsumed data amount in the n message partition information;

And under the condition that the unconsumed difference value meets a preset unconsumed threshold, adjusting a subscription mode of the consumer to finish the operation and maintenance of the distributed server cluster, wherein the consumer consumes the message data through the subscription mode.

According to an embodiment of the present disclosure, the above-described amount of unconsumed data is determined by:

acquiring consumption submission displacement and latest message displacement at the current moment in the message partition information;

and determining the amount of the unconsumed data according to the consumption submission displacement and the latest message displacement.

According to an embodiment of the present disclosure, the adjusting the subscription mode of the consumer includes:

acquiring a subscription mode of the consumption terminal, wherein the subscription mode comprises a specified partition subscription mode and an unspecified partition subscription mode;

and when the subscription mode of the consumer is the specified partition subscription mode, adjusting the specified partition subscription mode to be the unspecified partition subscription mode.

According to an embodiment of the present disclosure, adjusting the specified partition subscription mode to the unspecified partition subscription mode includes:

starting a new consumer, wherein the subscription mode of the new consumer is a unsubscribe mode, and the unsubscribe mode is used for canceling subscription of the consumer to the information partition information;

Under the condition of suspending the appointed partition subscription mode of the consumption terminal, suspending the unsubscribe mode of the new consumption terminal;

and calling an unspecified partition subscription mode as the subscription mode of the new consumption terminal so as to consume the message data of the message partition information by using the new consumption terminal.

According to an embodiment of the present disclosure, calculating an amount of message data within each of the above-described message partition information includes:

for each message partition information, acquiring an initial offset and an end offset in the message partition information at the current time;

and determining the message data amount in the message partition information according to the initial offset and the end offset.

According to an embodiment of the present disclosure, determining a data amount difference value according to a message data amount in a plurality of the above message partition information includes:

determining a maximum message data amount and a minimum message data amount according to the message data amount of each message partition information;

and determining the data quantity difference value according to the maximum message data quantity and the minimum message data quantity.

According to an embodiment of the present disclosure, adjusting the transmission mode of the production end includes:

Partition data according to the distributed server cluster;

determining a transmission mode of the production end according to the partition data, wherein the transmission mode comprises a specified partition transmission mode and an unspecified partition transmission mode;

when the transmission mode of the production side is a specified partition transmission mode, the specified partition transmission mode is adjusted to a non-specified partition transmission mode.

According to an embodiment of the present disclosure, adjusting the above specified partition transmission mode to a non-specified partition transmission mode includes:

and starting a new production end under the condition of suspending the uploading message data of the production end, wherein the new production end uploads the message data to the message partition information corresponding to the production end through a partition sending mode which is not specified.

According to an embodiment of the present disclosure, before calculating the message data amount, further includes:

acquiring cluster attribute information of the distributed server cluster, wherein the cluster attribute information comprises address information of p service nodes, connection information of a production end and the service nodes and connection information of a consumption end and the service nodes, and p is an integer greater than 0;

and performing connectivity test on the distributed server cluster according to the cluster attribute information to obtain a test result, so as to determine whether to calculate the message data volume according to the test result.

generating an operation and maintenance report according to difference information and adjustment information, wherein the difference information represents information for adjusting the transmission mode of the production end, and the difference value of the data quantity meets a preset difference threshold value;

and visually displaying the operation and maintenance report.

A second aspect of the present disclosure provides a partition equalization restoration device, including:

the cluster configuration manager is used for acquiring message attribute information uploaded to the distributed server cluster by a production end, wherein the message attribute information comprises m message topic information and n message partition information corresponding to the message topic information, m is an integer greater than 0, and n is an integer greater than 1;

a zone equalization detector for:

and the producer mode optimizer is used for adjusting the sending mode of the producer under the condition that the data quantity difference value meets a preset difference threshold value, wherein the producer uploads message data into the message partition information through the sending mode.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of operating the distributed server cluster described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method of operating a distributed server cluster.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above described method of operation and maintenance of a distributed server cluster.

According to the embodiment of the disclosure, by calculating the message data amount in each message partition information under the message subject information, whether the partition data imbalance problem exists in different message partition information under the message subject information is determined based on whether the data amount difference value between different message partition information is larger than the preset difference threshold value, so that whether the sending mode of a production end is changed is determined, abnormal running of the distributed server cluster caused by serious imbalance of the partition data due to continuous uploading of data in a later distributed server cluster is avoided, and the problems of low timeliness and high maintenance cost caused by manual checking whether the partition data imbalance exists after the message data with large-scale order magnitude is uploaded to the distributed server cluster can be avoided.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of an operation and maintenance method of a distributed server cluster according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a method of operation and maintenance of a distributed server cluster, in accordance with an embodiment of the present disclosure;

FIG. 3 schematically illustrates an architecture diagram of a distributed server cluster, according to an embodiment of the disclosure;

FIG. 4 schematically illustrates a flow chart of a method of operation and maintenance of a distributed server cluster according to another embodiment of the present disclosure;

FIG. 5 schematically illustrates a block diagram of a partition balancing restoration apparatus according to an embodiment of the present disclosure; and

fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement an operation and maintenance method of a distributed server cluster, according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

At present, there is no complete set of mature automated solution for multi-partition data imbalance of a distributed server cluster such as kafka in the field of financial science and technology, and the method completely depends on abnormal error report discovery of an application client, and in actual production operation, several disadvantages exist: firstly, the timeliness is low, the abnormality monitoring of the application client is completely relied on, the problem finding is late, and the processing is slow. Secondly, timeliness is low, no matched automatic solution exists at present, so that the overall problem solving time is long, and the current requirements for production emergency analysis cannot be met. Thirdly, the labor consumption is high, and at present, the automatic solution support is not available, so that the problem is detected by the human power, and the problem is analyzed and emergency is carried out.

In view of this, the embodiments of the present disclosure provide an operation and maintenance method, a partition balancing and recovering device, a device and a medium for a server cluster, which can be applied to the technical fields of finance and operation and maintenance. The operation and maintenance method comprises the following steps: acquiring message attribute information uploaded to a distributed server cluster by a production end, wherein the message attribute information comprises m message topic information and n message partition information corresponding to the message topic information; calculating the message data amount in each message partition information aiming at each message topic information; determining a data quantity difference value according to the message data quantity in the message partition information; and under the condition that the data quantity difference value meets a preset difference threshold value, adjusting a transmission mode of the production end, wherein the production end uploads the message data into the message partition information through the transmission mode.

In the technical scheme of the invention, the related user information (including but not limited to user personal information, user image information, user equipment information, such as position information and the like) and data (including but not limited to data for analysis, stored data, displayed data and the like) are information and data authorized by a user or fully authorized by all parties, and the processing of the related data such as collection, storage, use, processing, transmission, provision, disclosure, application and the like are all conducted according to the related laws and regulations and standards of related countries and regions, necessary security measures are adopted, no prejudice to the public welfare is provided, and corresponding operation inlets are provided for the user to select authorization or rejection.

It should be noted that, the operation and maintenance method and the partition balancing recovery device of the distributed server cluster provided by the present disclosure may be used in a financial field, for example, a financial institution such as a bank, and may also be used in any field other than the financial field, for example, a service provider such as a game, so the application field of the operation and maintenance method and the partition balancing recovery device of the distributed server cluster provided by the present disclosure is not limited.

Fig. 1 schematically illustrates an application scenario diagram of an operation and maintenance method of a distributed server cluster according to an embodiment of the present disclosure.

As shown in fig. 1, the application scenario 100 according to this embodiment may include a bank. The network 104 is the medium used to provide communication links between the terminal devices 101, 102, 103 and the distributed server cluster 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may interact with the distributed server cluster 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages, etc. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptops, desktops, etc., and the terminal devices 101, 102, 103 may be divided into a production end and a consumption end, the production end may upload message data into the distributed server cluster, and the consumption end may consume the message data in the distributed server cluster.

The distributed server cluster 105 may be a distributed server cluster providing various services, such as a background management distributed server cluster (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management distributed server cluster may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that, the operation and maintenance method of the distributed server cluster provided in the embodiment of the present disclosure may be performed by a partition balancing restoration device disposed in the distributed server cluster 105. Therefore, the partition balancing restoration apparatus provided by the embodiments of the present disclosure may be disposed in the distributed server cluster 105.

It should be understood that the number of servers in the terminal device, network, and distributed server cluster 105 in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 2 schematically illustrates a flow chart of an operation and maintenance method of a distributed server cluster according to an embodiment of the disclosure. Fig. 3 schematically illustrates an architecture diagram of a distributed server cluster, according to an embodiment of the disclosure.

As shown in fig. 2 and 3, the operation and maintenance method of the distributed server cluster applied to the partition balancing restoration apparatus of this embodiment includes operations S210 to S230.

In operation S210, obtaining message attribute information uploaded to a distributed server cluster by a production end, where the message attribute information includes m message topic information and n message partition information corresponding to the message topic information, m is an integer greater than 0, and n is an integer greater than 1;

in operation S220, for each message topic information, calculating a message data amount within each message partition information;

in operation S230, a data amount difference value is determined according to the message data amounts within the plurality of message partition information;

in operation S240, if the data amount difference value meets the preset difference threshold, the transmission mode of the production end is adjusted, where the production end uploads the message data into the message partition information through the transmission mode.

According to an embodiment of the present disclosure, the distributed server cluster may be a Kafka distributed message service cluster. The partition balancing restoration means may be arranged within the distributed server cluster.

According to embodiments of the present disclosure, a production end may refer to a device that a producer uploads message data to a distributed server cluster. Message topic information is a logical concept, which is the kind of message. One message topic information may have a plurality of message partition information components. The producer publishes the message (i.e., message data) to the message subject information from which the consumer reads the message, e.g., 1 message subject information.

According to embodiments of the present disclosure, message partition information parttoin is a physical concept that divides message subject information into a plurality of partitions (i.e., message partition information), each having a unique identifier. The multi-partition is to promote concurrency, and can optimize production and consumption performance, for example, 2 message partition information is provided.

According to the embodiment of the disclosure, the sending mode refers to that a producer can issue a message to a certain designated message partition information of the message topic information through a production end, and can also select a default random uniform message to be issued to all message partition information of the message topic information.

According to the embodiment of the disclosure, in the process that the production end uploads the message data to the distributed server cluster in real time, the partition balancing recovery device is utilized to acquire the message data volume in each message partition information in real time (or at fixed time), and then the data volume difference value between the message data volumes in different message partition information is determined.

According to the embodiment of the disclosure, if the data amount difference value is greater than a certain preset difference threshold, the distributed server cluster may be considered to have a problem of unbalanced partition data, where the preset difference threshold may be specifically set according to an actual requirement or a data amount of the message data, for example, may be 2%. At this time, the transmission mode of the production end should be modified, for example, the transmission mode of the specified message partition information can be adjusted to be a transmission mode of randomly and uniformly issued messages, and the modification of the transmission mode can enable the distributed server cluster to transmit the message data into a plurality of message partition information in the message subject information based on the distribution rule of the distributed server cluster after receiving the message data uploaded by the production end, thereby being beneficial to improving the current problem of unbalanced partition data.

Fig. 4 schematically illustrates a flow chart of a method of operation and maintenance of a distributed server cluster according to another embodiment of the present disclosure.

As shown in fig. 4, the operation and maintenance method further includes operations S410 to S430.

In operation S410, if the data amount difference value does not meet the preset difference threshold, acquiring cluster attribute information of the distributed server cluster, where the cluster attribute information includes unconsumed data amounts of message data in the message partition information by different consuming ends;

In operation S420, an unconsumed difference value is determined according to the unconsumed data amount in the n message partition information;

in operation S430, in the case that the unconsumed difference value satisfies the preset unconsumed threshold, the subscription mode of the consumer is adjusted, where the consumer consumes the message data through the subscription mode.

According to the embodiment of the disclosure, if the difference values of the data amounts are smaller than the preset difference threshold, it is indicated that the problem of unbalanced partition data does not exist in the n message partition information under the message subject information at this time, and at this time, whether the problem of unbalanced consumption exists needs to be continuously judged, and the unbalanced consumption is caused by that the consumption end consumes the message data to cause that the difference amount of the message data which is not consumed in the n message partition information is larger, and the operation of the distributed server cluster may be abnormal due to unbalanced consumption.

According to embodiments of the present disclosure, the larger the value of the unconsumed data amount LAG, the more message data is illustrated as not being consumed. If the production rate of message data far exceeds the consumption rate, the LAG value will become large, representing a consumption LAG.

According to the embodiment of the disclosure, for each message topic information, statistics is performed on the amount of unconsumed data in each message partition information in the message topic information, unconsumed difference values between different message partition information are determined, and if any one of a plurality of unconsumed difference values is greater than a preset unconsumed difference value, the preset unconsumed difference value can be specifically set according to actual requirements, for example, 5%, then it can be determined that the message topic information has a problem of unbalanced consumption, so that at this time, a subscription mode of a consuming end needs to be adjusted to change the phenomenon of unbalanced consumption.

According to the embodiment of the disclosure, the unconsumed difference value between different message partition information is determined according to the unconsumed data amount in different message partition information in the message theme information, whether the problem of unbalanced consumption exists in the different message partition information under the message theme information is determined based on whether the unconsumed difference value between the different message partition information is larger than the preset unconsumed difference value, so that whether the subscription mode of a consumer terminal is changed is determined, abnormal running of the distributed server cluster caused by serious unbalanced consumption due to continuous uploading of data in a later distributed server cluster is avoided, and the problems of low timeliness and high maintenance cost caused by manual checking whether the unbalanced consumption exists after the message data in a large-scale order is uploaded to the distributed server cluster can be avoided.

According to an embodiment of the present disclosure, the amount of unconsumed data is determined by:

acquiring consumption submission displacement and latest message displacement at the current moment in the message partition information; and determining the amount of unconsumed data according to the consumption submission displacement and the latest message displacement.

According to the embodiment of the disclosure, the consumption submission displacement and the latest message displacement are automatically recorded by the distributed server cluster according to the consumption condition of the consumption end, wherein the consumption submission displacement represents message data which is already consumed by the consumption end at the current moment, and the latest message displacement represents message data to be consumed which is required to be consumed by the consumption end, so that the amount of unconsumed data which is not consumed yet can be determined according to the consumption submission displacement and the latest message displacement.

According to an embodiment of the present disclosure, adjusting a subscription mode of a consumer includes:

acquiring a subscription mode of a consumption terminal, wherein the subscription mode comprises a designated partition subscription mode and an unspecified partition subscription mode; and in the case that the subscription mode of the consumption side is the appointed partition subscription mode, adjusting the appointed partition subscription mode to an unspecified partition subscription mode.

According to the embodiment of the disclosure, the designated message partition information in the message topic information can be subscribed by the designated partition subscription mode assignment, and one or more message partition information under one or more message topic information can be subscribed by the unspecified partition subscription mode subscription.

According to the embodiment of the disclosure, with the command line tool in the consumer subscription pattern analyzer, detailed information of the consumer may be returned, including subscription patterns of the topic information topic of the subscription message of the consumer, and if the subscription patterns used by the consumer are confirmed to be the specified partition subscription patterns, the specified partition subscription patterns may be adjusted to be unspecified partition subscription patterns.

According to the embodiment of the disclosure, when consumption imbalance occurs, the designated partition subscription mode of the consumption terminal is automatically adjusted to be the unspecified partition subscription mode, so that the problem of consumption imbalance of the current distributed server cluster is solved, and the occurrence of abnormal operation of the distributed server cluster caused by consumption imbalance in the later period is avoided.

According to an embodiment of the present disclosure, adjusting a specified partition subscription mode to an unspecified partition subscription mode includes: starting a new consumption terminal, wherein the subscription mode of the new consumption terminal is a unsubscribe mode, and the unsubscribe mode is used for canceling the subscription of the consumption terminal to the information partition information; suspending the unsubscribe mode of the new consumer under the condition of suspending the appointed subarea subscription mode of the consumer; and calling the unspecified partition subscription mode as a subscription mode of a new consumption end so as to consume the message data of the message partition information by using the new consumption end.

According to embodiments of the present disclosure, upon determining that a specified partition subscription mode assignment is used at the consumer, the specified partition subscription mode assignment needs to be adjusted to an unspecified partition subscription mode subset. The method is that a new consumer is started first, and the new consumer uses an unsubscribe mode, which is used for canceling the subscription of the original consumer to the message topic information topic. And secondly stopping the consumer thread subscribed by the assigned partition subscription mode assignment of the original consumer terminal. The new consumer that invoked the unsubscribe mode unsubscribe is then stopped. And finally, starting a new consumption terminal which calls a subscription mode subscore of an unspecified partition, automatically adding the new consumption terminal into a consumer group, and starting to consume the message data.

According to an embodiment of the present disclosure, calculating an amount of message data within each message partition information includes: for each message partition information, acquiring an initial offset and an end offset in the message partition information at the current moment; and determining the message data amount in the message partition information according to the initial offset and the end offset.

According to the embodiment of the disclosure, the initial offset and the end offset are automatically recorded by the distributed server cluster according to the condition that the production end uploads the message data, for example, for certain message partition information, the message partition information can be given an identifier of "0" when the message data uploaded by the production end is stored for the first time, the identifier can represent the initial offset, and the message data can be given an identifier of "n" when the message data is stored for the nth time at the current moment, and the identifier can represent the end offset.

According to the embodiment of the disclosure, a partition balance detector connected with a distributed server cluster is utilized to query all existing message topic information topic of the distributed server cluster, the message topic information topic is returned to the partition balance detector as a set, then all message topic information topic is subscribed and consumed, an initial offset and an end offset of each message partition information are obtained, the message data of each message partition information of the message topic information topic is calculated through the end offset and the initial offset difference, difference comparison is carried out according to the message data of each message partition information, and if the message data of a certain message partition information and the message data of the rest message partition information float up and down by more than 2%, the condition that the message topic information topic of the distributed server cluster has unbalanced partition data is judged.

According to the embodiment of the disclosure, by determining the message data amount in the message partition information at the current moment according to the initial offset and the final offset in real time, the problem of inaccurate message data amount calculation caused by the fact that the message partition information is endowed with the mark by the message partition information or the first message data in the message partition information is not 0 due to other factors can be avoided.

According to an embodiment of the present disclosure, determining a data amount difference value from a message data amount within a plurality of message partition information includes: determining a maximum message data amount and a minimum message data amount according to the message data amount of each message partition information; and determining a data quantity difference value according to the maximum message data quantity and the minimum message data quantity.

According to the embodiment of the disclosure, for n message partition information in the message subject information, a maximum message data amount and a minimum message data amount in the n message partition information are first determined, then a difference is made based on the maximum message data amount and the minimum message data amount, and the difference value is compared with the maximum message data amount to determine a data amount difference value.

Note that, the calculation of the unconsumed difference value is the same as the calculation of the data amount difference value, and a detailed description is omitted here.

According to an embodiment of the present disclosure, adjusting a transmission mode of a production end includes: partition data according to the distributed server cluster; determining a transmission mode of a production end according to the partition data, wherein the transmission mode comprises a specified partition transmission mode and an unspecified partition transmission mode; when the transmission mode of the production side is the specified partition transmission mode, the specified partition transmission mode is adjusted to the non-specified partition transmission mode.

According to the embodiment of the disclosure, partition data, such as log files of time, path and the like, when each production end uploads certain message partition information, are automatically recorded in the running process of the distributed server cluster.

According to the embodiment of the disclosure, the sending mode of the production end can be determined by analyzing the partition data, when the problem of unbalance of the partition data exists at the current moment is determined, if the sending mode of the production end is the designated partition sending mode, the problem of unbalance of the partition data is more serious in the continuous uploading of the later production end if the mode is not regulated, so that after the message data is uploaded to the distributed server cluster by the production end, the distributed server cluster can determine which message partition information the message data is stored in according to own rules by regulating the designated partition sending mode to a regulation mode of not designating the partition sending mode.

According to an embodiment of the present disclosure, adjusting the specified partition transmission mode to the non-specified partition transmission mode includes: and starting a new production end under the condition of suspending the uploading of the message data of the production end, wherein the new production end uploads the message data to the message partition information corresponding to the production end by not designating a partition sending mode.

According to the embodiment of the disclosure, the production end which needs to be optimized and adjusted at present is stopped, and then a new production end which uses a non-appointed partition sending mode is started to perform the work of the original production end, namely, the message data is uploaded to the same message topic information topic.

According to an embodiment of the present disclosure, before calculating the message data amount, further comprising: acquiring cluster attribute information of a distributed server cluster, wherein the cluster attribute information comprises address information of p service nodes, connection information of a production end and the service nodes and connection information of a consumption end and the service nodes, and p is an integer larger than 0; and performing connectivity test on the distributed server cluster according to the cluster attribute information to obtain a test result, and determining whether to calculate the message data volume according to the test result.

According to embodiments of the present disclosure, a distributed server cluster first needs to be tested for connectivity before it begins to operate to ensure that the distributed server cluster is able to communicate with all production and consumer ends properly.

According to the embodiment of the disclosure, in the connectivity test process, connectivity verification needs to be performed on any one production end and consumption end of the distributed server cluster, and after all verification passes, judgment of partition data imbalance or consumption imbalance is performed.

According to an embodiment of the present disclosure, the operation and maintenance method further includes: generating an operation and maintenance report according to the difference information and the adjustment information, wherein the difference information characterizes that the difference value of the data quantity meets a preset difference threshold value, and the adjustment information characterizes information for adjusting the transmission mode of the production end; the operation and maintenance report is visually displayed.

According to the embodiment of the present disclosure, the difference information may include not only a determination result of whether or not there is a partition data imbalance, but also a determination result of whether or not there is a consumption imbalance, and thus the adjustment information may also include adjustment information of a subscription pattern made for the consumption imbalance.

According to the embodiment of the disclosure, the operation and maintenance report is generated by the judging result and the adjusting information, and then the operation and maintenance report is displayed to the corresponding operation and maintenance department, wherein the operation and maintenance report can comprise a plurality of sub-reports, and each sub-report can be sent to different operation and maintenance departments in the displaying process so that the different operation and maintenance departments can perform self-checking on the judging result and the adjusting information.

Fig. 5 schematically illustrates a block diagram of a partition balancing restoration apparatus according to an embodiment of the present disclosure.

As shown in fig. 5, the partition balancing restoration apparatus 500 of this embodiment includes a cluster configuration manager 510, a partition balancing detector 520, and a producer pattern optimizer 530.

A cluster configuration manager 510, configured to obtain message attribute information uploaded to a distributed server cluster by a production end, where the message attribute information includes m message topic information and n message partition information corresponding to the message topic information, m is an integer greater than 0, and n is an integer greater than 1;

a zone equalization detector 520 for: calculating the message data amount in each message partition information aiming at each message topic information; determining a data quantity difference value according to the message data quantity in the message partition information;

and the producer mode optimizer 530 is configured to adjust a transmission mode of the producer in case that the difference value of the data amount satisfies a preset difference threshold, where the producer uploads the message data to the message partition information according to the transmission mode.

According to an embodiment of the present disclosure, the partition balancing restoration apparatus 500 further includes a consumption pile-up detector, a consumer subscription pattern optimizer.

The cluster configuration manager is further configured to obtain cluster attribute information of the distributed server cluster when the data volume difference value does not meet a preset difference threshold, where the cluster attribute information includes unconsumed data volumes of message data in the message partition information by different consuming ends;

the consumption accumulation detector is used for determining an unconsumed difference value according to unconsumed data volume in the n message partition information;

and the consumer subscription mode optimizer is used for adjusting the subscription mode of the consumer terminal under the condition that the unconsumed difference value meets the preset unconsumed threshold, wherein the consumer terminal consumes the message data through the subscription mode.

According to an embodiment of the present disclosure, the amount of unconsumed data is determined by the acquisition unit, the determination unit:

the acquisition unit is used for acquiring consumption submission displacement and latest message displacement at the current moment in the message partition information;

and the determining unit is used for determining the amount of unconsumed data according to the consumption submission displacement and the latest message displacement.

According to an embodiment of the disclosure, a consumer subscription pattern optimizer includes a first acquisition module, a first adjustment module.

The first acquisition module is used for acquiring a subscription mode of a consumer, wherein the subscription mode comprises a designated partition subscription mode and an unspecified partition subscription mode;

the first adjusting module is used for adjusting the appointed partition subscription mode to an unspecified partition subscription mode under the condition that the subscription mode of the consumption terminal is the appointed partition subscription mode.

According to an embodiment of the disclosure, the adjustment module includes a start unit, a pause unit, and a call unit.

The starting unit is used for starting a new consumption end, wherein the subscription mode of the new consumption end is a unsubscribe mode, and the unsubscribe mode is used for canceling the subscription of the consumption end to the information partition information;

a suspension unit, configured to suspend an unsubscribe mode of the new consumer in a case of suspending a specified partition subscription mode of the consumer;

and the calling unit is used for calling the subscription mode of the unspecified partition as the subscription mode of the new consumption terminal so as to consume the message data of the message partition information by using the new consumption terminal.

According to an embodiment of the present disclosure, the partition balancing detector 520 includes a second acquisition module, a first determination module.

The second acquisition module is used for acquiring initial offset and end offset in the message partition information at the current moment according to each message partition information;

And the first determining module is used for determining the message data amount in the message partition information according to the initial offset and the end offset.

According to an embodiment of the present disclosure, the partition balancing detector 520 further includes a second determination module, a third determination module.

The second determining module is used for determining the maximum message data amount and the minimum message data amount according to the message data amount of each message partition information;

and the third determining module is used for determining a data quantity difference value according to the maximum message data quantity and the minimum message data quantity.

According to an embodiment of the present disclosure, the producer pattern optimizer 530 includes a third acquisition module, a fourth determination module, a second adjustment module.

The third acquisition module is used for acquiring partition data of the distributed server cluster;

a fourth determining module, configured to determine a transmission mode of the production end according to the partition data, where the transmission mode includes a specified partition transmission mode and a non-specified partition transmission mode;

and the second adjusting module is used for adjusting the appointed partition sending mode to the non-appointed partition sending mode when the sending mode of the production side is the appointed partition sending mode.

According to an embodiment of the present disclosure, the second adjustment module comprises an adjustment unit.

And the adjusting unit is used for starting a new production end under the condition of suspending the uploading of the message data of the production end, wherein the new production end uploads the message data to the message partition information corresponding to the production end through a partition sending mode which is not specified.

According to an embodiment of the present disclosure, the partition balancing restoration apparatus 500 further includes a connectivity tester.

The cluster configuration manager is further configured to obtain cluster attribute information of the distributed server cluster, where the cluster attribute information includes address information of p service nodes, connection information of a production end and the service nodes, and connection information of a consumption end and the service nodes, and p is an integer greater than 0;

and the connectivity tester is used for carrying out connectivity test on the distributed server clusters according to the cluster attribute information to obtain test results so as to determine whether to calculate the message data volume according to the test results.

According to an embodiment of the present disclosure, the partition balancing restoration apparatus 500 further includes a generating module and a displaying module.

The generation module is used for generating an operation and maintenance report according to the difference information and the adjustment information, wherein the difference information characterizes that the difference value of the data quantity meets a preset difference threshold value, and the adjustment information characterizes information for adjusting the transmission mode of the production end;

And the display module is used for visually displaying the operation and maintenance report.

Any of the plurality of devices (or modules described above) in cluster configuration manager 510, partition balancing detector 520, producer pattern optimizer 530 may be combined in one device (or module) or any of the plurality of devices (or modules) may be split into multiple devices (or modules) according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the devices (or modules) may be combined with at least some of the functionality of other devices (or modules) and implemented in one device (or module). At least one of cluster configuration manager 510, partition balancing detector 520, producer pattern optimizer 530 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), programmable Logic Array (PLA), system on a chip, system on a substrate, system on a package, application Specific Integrated Circuit (ASIC), or in hardware or firmware, in any other reasonable manner of integrating or packaging circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware, according to embodiments of the present disclosure. Alternatively, at least one of cluster configuration manager 510, partition balancing detector 520, producer pattern optimizer 530 may be at least partially implemented as computer program modules which, when executed, perform the corresponding functions.

As shown in fig. 6, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. The processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or the RAM 603. Note that the program may be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 600 may also include an input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604. The electronic device 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 602 and/or RAM 603 and/or one or more memories other than ROM 602 and RAM 603 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code, when executed in a computer system, causes the computer system to implement the methods of operation and maintenance provided by embodiments of the present disclosure.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and/or installed from the removable medium 611. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. An operation and maintenance method of a distributed server cluster is applied to a partition equilibrium recovery device, and the method comprises the following steps:

calculating the message data amount in each message partition information aiming at each message topic information;

and under the condition that the data quantity difference value meets a preset difference threshold, adjusting a sending mode of the production end, wherein the production end uploads message data into the message partition information through the sending mode.

2. The method of claim 1, further comprising:

And under the condition that the unconsumed difference value meets a preset unconsumed threshold, adjusting a subscription mode of the consumer, wherein the consumer consumes the message data through the subscription mode.

3. The method of claim 2, wherein the amount of unconsumed data is determined by:

4. The method of claim 2, wherein adjusting the subscription mode of the consumer comprises:

and under the condition that the subscription mode of the consumption terminal is the appointed partition subscription mode, adjusting the appointed partition subscription mode to be the unspecified partition subscription mode.

5. The method of claim 4, wherein adjusting the specified partition subscription mode to the unspecified partition subscription mode comprises:

starting a new consumption terminal, wherein the subscription mode of the new consumption terminal is a unsubscribe mode, and the unsubscribe mode is used for canceling subscription of the consumption terminal to the information partition information;

6. The method of claim 1, wherein calculating the amount of message data within each of the message partition information comprises:

and determining the message data amount in the message partition information according to the initial offset and the tail offset.

7. The method of claim 1, wherein determining a data volume difference value from the message data volumes within the plurality of message partition information comprises:

8. The method of claim 1, wherein adjusting the transmission mode of the production end comprises:

Obtaining partition data of the distributed server cluster;

and when the transmission mode of the production side is a specified partition transmission mode, adjusting the specified partition transmission mode to a non-specified partition transmission mode.

9. The method of claim 8, wherein adjusting the specified partition transmission mode to a non-specified partition transmission mode comprises:

and under the condition of suspending the uploading of the message data of the production end, starting a new production end, wherein the new production end uploads the message data to the message partition information corresponding to the production end through a partition sending mode which is not specified.

10. The method of claim 1, wherein prior to calculating the message data amount, further comprising:

And performing connectivity test on the distributed server cluster according to the cluster attribute information to obtain a test result, and determining whether to calculate the message data volume according to the test result.

11. The method of claim 1, further comprising:

generating an operation and maintenance report according to difference information and adjustment information, wherein the difference information represents that the data quantity difference value meets a preset difference threshold value, and the adjustment information represents information for adjusting the transmission mode of the production end;

the operation and maintenance report is visually displayed.

12. A partition equalization restoration device, comprising:

a zone equalization detector for:

And the producer mode optimizer is used for adjusting the sending mode of the producer under the condition that the data quantity difference value meets a preset difference threshold, wherein the producer uploads message data into the message partition information through the sending mode.

13. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-11.

14. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-11.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.