CN117407158A

CN117407158A - Cluster capacity expansion method and device, electronic equipment and storage medium

Info

Publication number: CN117407158A
Application number: CN202311294592.8A
Authority: CN
Inventors: 王志龙; 黄毓铭; 赵子颖; 邓琛; 丁家文
Original assignee: Tianyi Digital Life Technology Co Ltd
Current assignee: Tianyi Digital Life Technology Co Ltd
Priority date: 2023-10-08
Filing date: 2023-10-08
Publication date: 2024-01-16

Abstract

The invention discloses a cluster capacity expansion method, a cluster capacity expansion device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring data volume change information of message middleware and resource consumption data of cluster machines; further, according to the resource consumption data, combining the data quantity information to obtain resource consumption prediction data; when the resource consumption prediction data reaches a preset threshold value, determining capacity expansion information based on a resource consumption target value; and carrying out cluster capacity expansion according to the capacity expansion information. According to the embodiment of the invention, through the logic analysis of the information of the change of the data in the message intermediate data, the resource consumption prediction data is obtained by combining the resource consumption data already used by the cluster machine, and further the determination of the capacity expansion information is realized based on the resource consumption prediction data, so that the self-adaptive capacity expansion of the cluster can be rapidly and timely carried out, the capacity expansion cost is avoided, the capacity expansion efficiency of the distributed cloud cluster is improved, and the method and the device can be widely applied to the technical field of data processing.

Description

Cluster capacity expansion method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a cluster expansion method, a device, an electronic device, and a storage medium.

Background

With the continuous development of the service scale of the product, the requirements on data storage and calculation of the distributed cloud clusters are increased, the original cloud clusters, such as Hadoop, elasticsearch, cannot meet the requirement of rapid service growth, and when the use of the distributed cloud clusters reaches the upper limit of the design capacity of the cloud clusters, the capacity of the cloud clusters needs to be expanded.

Whether the capacity expansion is required is generally judged by monitoring the load of the cluster, the capacity expansion is carried out by the load change of the cluster, the capacity expansion information cannot be effectively determined, the capacity expansion is easy to be overlarge or the capacity expansion is too small and needs to be carried out for a plurality of times, so that the capacity expansion cost is high, and the capacity expansion of the existing big data cluster is inflexible.

Disclosure of Invention

The present invention aims to solve at least one of the technical problems in the related art to some extent. Therefore, the invention provides a cluster capacity expansion method, a cluster capacity expansion device, electronic equipment and a storage medium, which can efficiently and accurately perform cluster capacity expansion.

In one aspect, an embodiment of the present invention provides a cluster expansion method, including:

acquiring data volume change information of the message middleware and resource consumption data of the cluster machine;

obtaining resource consumption prediction data according to the resource consumption data and combining the data quantity change information;

When the predicted data of the resource consumption reaches a preset threshold value, determining capacity expansion information based on a target value of the resource consumption;

and carrying out cluster capacity expansion according to the capacity expansion information.

Optionally, acquiring the data volume change information of the message middleware and the resource consumption data of the cluster machine includes:

acquiring the data quantity of production and consumption of all subjects of the message middleware in a preset time period as the data quantity change information of the message middleware;

and obtaining disk capacity consumption data of the cluster machine and CPU utilization rate of the cluster machine.

Optionally, the method further comprises:

based on the stress test of the cluster machine, obtaining a relation data list of data quantity change information and newly-added resource consumption data, and determining preset parameters; wherein the preset parameter includes at least one of a preset threshold value and a resource consumption target value.

Optionally, obtaining the resource consumption prediction data according to the resource consumption data and the data quantity information, including:

based on the relation data list, obtaining newly added resource consumption data according to the data quantity change information; the relation data list is obtained based on a pressure test of the cluster machine;

and obtaining resource consumption prediction data according to the resource consumption data and the newly added resource consumption data.

Optionally, the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; when the predicted data of the resource consumption reaches a preset threshold, determining capacity expansion information based on the target value of the resource consumption, including:

when the predicted data of the disk capacity consumption reaches a first threshold value, acquiring the total space of the disk capacity, and subtracting the total space of the disk capacity from the ratio of the data of the disk capacity consumption to the target value of the disk capacity consumption to obtain the capacity expansion information of the disk;

and/or when the CPU utilization rate reaches a second threshold value, acquiring the number of machines of the cluster machines, and acquiring the machine capacity expansion information according to the number of machines and the preset proportion.

Optionally, the capacity expansion information includes disk capacity expansion information and/or machine capacity expansion information; performing cluster capacity expansion according to capacity expansion information, including:

performing disk capacity expansion on the cluster machine according to the disk capacity expansion information;

and/or configuring the new machine according to the machine capacity expansion information, adding the configured new machine into the cluster machine, and completing the capacity expansion of the cluster machine.

Optionally, the method further comprises:

acquiring the capacity expansion operation result of the cluster machine after completing the cluster capacity expansion;

and when the capacity expansion operation result reaches the expected operation range, completing the capacity expansion of the cluster, otherwise, carrying out the capacity expansion of the cluster again based on the capacity expansion operation result.

In another aspect, an embodiment of the present invention provides a cluster expansion device, including:

the first module is used for acquiring the data volume change information of the message middleware and the resource consumption data of the cluster machine;

the second module is used for obtaining resource consumption prediction data according to the resource consumption data and combining the data quantity variation information;

a third module, configured to determine expansion information based on the resource consumption target value when the resource consumption prediction data reaches a preset threshold;

and the fourth module is used for carrying out cluster capacity expansion according to the capacity expansion information.

Optionally, the first module is specifically configured to:

Optionally, the apparatus further comprises:

a fifth module, configured to obtain a relational data list of the data amount change information and the newly added resource consumption data based on a stress test on the clustered machine, and determine a preset parameter; wherein the preset parameter includes at least one of a preset threshold value and a resource consumption target value.

Optionally, the second module is specifically configured to:

Optionally, the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; the third module is specifically configured to:

Optionally, the capacity expansion information includes disk capacity expansion information and/or machine capacity expansion information, and the fourth module is specifically configured to:

Optionally, the apparatus further comprises:

a sixth module, configured to obtain a capacity expansion operation result of the clustered machine after completing the capacity expansion of the cluster;

and a seventh module, configured to complete cluster capacity expansion when the capacity expansion operation result reaches the expected operation range, and if not, perform cluster capacity expansion again based on the capacity expansion operation result.

In another aspect, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used for storing programs; the processor executes the program to realize the cluster expansion method.

In another aspect, an embodiment of the present invention provides a computer storage medium in which a program executable by a processor is stored, where the program executable by the processor is configured to implement the cluster expansion method described above when executed by the processor.

The embodiment of the invention obtains the data volume change information of the message middleware and the resource consumption data of the cluster machine; further, according to the resource consumption data, combining the data quantity change information to obtain resource consumption prediction data; when the predicted data of the resource consumption reaches a preset threshold value, determining capacity expansion information based on a target value of the resource consumption; and carrying out cluster capacity expansion according to the capacity expansion information. According to the embodiment of the invention, through the logic analysis of the information of the change of the data in the message intermediate, the resource consumption prediction data is obtained by combining the resource consumption data already used by the cluster machine, and further the capacity expansion information is determined based on the resource consumption prediction data, so that the self-adaptive capacity expansion of the cluster can be rapidly and timely carried out, the capacity expansion cost is avoided, and the capacity expansion efficiency of the distributed cloud cluster is improved.

Drawings

The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.

FIG. 1 is a schematic diagram of an implementation environment for performing cluster expansion according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a cluster expansion method according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart of acquiring data amount information and resource consumption data according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart of obtaining predicted data of resource consumption according to an embodiment of the present invention;

fig. 5 is a schematic flow chart of determining capacity expansion information according to an embodiment of the present invention;

fig. 6 is a schematic overall flow chart of a cluster capacity expansion method combined with capacity expansion result judgment according to an embodiment of the present invention;

fig. 7 is an overall flow schematic diagram of a cluster capacity expansion method according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a cluster expansion device according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 10 is a block diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

It should be noted that although functional block diagrams are depicted as block diagrams, and logical sequences are shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the block diagrams in the system. The terms first/S100, second/S200, and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

It can be understood that the cluster expansion method provided by the embodiment of the invention can be applied to any computer device with data processing and computing capabilities, and the computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or is a server cluster or a distributed system formed by a plurality of physical servers, or is a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network ), basic cloud computing services such as big data and artificial intelligence platforms, and the like. Alternatively, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like, but is not limited thereto.

FIG. 1 is a schematic view of an embodiment of the invention. Referring to fig. 1, the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected through a network in a wireless or wired mode to complete data transmission and exchange.

The server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

In addition, server 101 may also be a node server in a blockchain network. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like.

The terminal 102 may be, but is not limited to, a smart phone, tablet, notebook, desktop, smart box, smart watch, etc. The terminal 102 and the server 101 may be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present invention.

Based on the implementation environment shown in fig. 1, the embodiment of the present invention provides a cluster expansion method, and the description below uses the application of the cluster expansion method in the server 101 as an example, and it can be understood that the cluster expansion method may also be applied in the terminal 102.

Referring to fig. 2, fig. 2 is a flowchart of a cluster expansion method applied to a server according to an embodiment of the present invention, where an execution body of the cluster expansion method may be any one of the foregoing computer devices (including a server or a terminal). Referring to fig. 2, the method includes the steps of:

s100, acquiring data volume change information of message middleware and resource consumption data of cluster machines;

it should be noted that, in some embodiments, as shown in fig. 3, step S100 may include: s101, acquiring the data quantity of production and consumption of all subjects of the message middleware in a preset time period as the data quantity information of the message middleware; and S102, acquiring disk capacity consumption data of the clustered machine, and acquiring CPU utilization rate of the clustered machine.

In some embodiments, the data volume change information of the message middleware and the resource consumption data of the cluster machine may be acquired by the client agent, where the flow steps acquired by the client agent are as follows:

firstly, a agent system needs to be deployed in each machine of the clustered machine, and the change of the data volume of the message middleware is mainly collected, wherein the change comprises information of the load, the memory and the disk use of the clustered machine, and the running state information of the clustered software, such as information of the production data volume of each 1 minute, 15 minutes and 60 minutes of Partition on each Topic of all topics of the message middleware, is collected.

Where Topic is the subject of a message, it is understood that the classification of a message, multiple topics may be created on the message middleware. Message system-pooling in publish-subscribe mode-different producers send messages to topic and MQ server publishes messages to different subscribers to realize the broadcasting of messages. topic is a logical concept, and a topic may contain multiple parts, which are physical concepts, and kafka stores the parts on a brooker disk.

And collecting and storing the acquired data, reporting the use information of each machine and the running state information of the software in the cluster to a database in real time through an interface by a client, and mainly analyzing by a subsequent analysis module to realize the subsequent cluster capacity expansion step.

S200, obtaining resource consumption prediction data according to the resource consumption data and combining data quantity change information;

in some embodiments, the method may further include: based on the stress test of the cluster machine, obtaining a relation data list of data quantity change information and newly-added resource consumption data, and determining preset parameters; wherein the preset parameter includes at least one of a preset threshold value and a resource consumption target value.

In some embodiments, as shown in fig. 4, step S200 may include: s201, based on a relation data list, obtaining newly added resource consumption data according to data quantity change information; the relation data list is obtained based on a pressure test of the cluster machine; s202, obtaining resource consumption prediction data according to the resource consumption data and the newly added resource consumption data.

In some embodiments, the resource consumption prediction data may be obtained by an analysis module, where the flow implemented by the analysis module includes the following steps: the analysis module collects data quantity of message middleware production and consumption and machine resource consumption data, including CPU load, memory use, disk use and other operation parameters, and obtains data of throughput, consumption, distributed cluster resource consumption and the like tested in each stage from message middleware data quantity change and distributed cluster resource consumption by performing pressure test on cluster machines before online, so as to obtain a maximum value and normal value data list which can be born by message middleware production and consumption data quantity and corresponding clusters, and generally considers problems of redundancy, system space and the like, and meanwhile, the space required by the disks in calculation is considered, and the total disk capacity redundancy is generally 30%, so that the data quantity of message middleware production and consumption is added with the current load value of the clusters, the predicted cluster disk capacity exceeds 70%, or the CPU utilization rate reaches more than 80%, and the capacity is expanded.

S300, when the predicted data of the resource consumption reaches a preset threshold value, determining capacity expansion information based on a target value of the resource consumption;

it should be noted that, the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target includes a disk capacity consumption target, and in some embodiments, as shown in fig. 5, step S300 may include:

s301, when the predicted data of the disk capacity consumption reaches a first threshold value, acquiring the total space of the disk capacity, subtracting the total space of the disk capacity from the ratio of the data of the disk capacity consumption to a target value of the disk capacity consumption, and acquiring the capacity expansion information of the disk;

and/or S302, when the CPU utilization rate reaches a second threshold value, acquiring the number of machines of the cluster machine, and acquiring the machine capacity expansion information according to the number of machines and the preset proportion.

In some embodiments, the analysis module based on the steps determines the predicted resource consumption data and the corresponding preset threshold, so that the following flow logic may be implemented through the analysis module:

the analysis module can judge whether the capacity expansion mode judged by the analysis module comprises disk capacity expansion and CPU capacity expansion according to the increase of the production and consumption data volume of the message middleware and the logic judgment of whether the cluster computing resources and the storage resources are expanded, the cost is considered during the capacity expansion, the cost is generally considered to be a reasonable value after the capacity expansion, and the resource waste caused by too much one-time capacity expansion is avoided, and the specific contents are as follows:

Disk expansion model: the method comprises the steps of carrying out a disk total capacity pressure test before online, obtaining the number of cluster disk storage resources required by the change of different data volumes according to the relation between the data volume produced and consumed by a message middleware of a cluster and the cluster disk utilization rate, namely, a disk model calculation formula, analyzing the data volume increment of the message middleware by an analysis module, collecting and analyzing the data volume change in the time period of 1 minute, 15 minutes and 60 minutes of each partition of Topic, and simultaneously calculating whether the generated data volume in the time period exceeds the capacity expansion condition of the existing cluster, namely, comprehensively evaluating whether the data volume change of all Topic subject data volumes of the message middleware and the index of the cluster operation current situation exceed the pre-warning value born by the cluster, for example: the data volume of the message middleware is increased, the daily estimated increased cluster disk capacity is obtained according to the data volume produced and consumed by the message middleware, the estimated increased disk capacity of the data in a week is obtained in consideration of the capacity expansion cost, and if the estimated increased disk capacity of the cluster in a week plus the current cluster disk capacity exceeds 70%, the cluster needs to be expanded. The running level of the cluster after capacity expansion is normal (the total disk capacity utilization rate of the cluster after capacity expansion is not more than 50% and is normal, and the cluster can be changed into a proper value according to actual needs).

The disk model calculation formula: u/(f+x) =y, space (U) has been used before expansion, total space (F) before expansion, this time the storage space (X) to be newly added, total space usage (y=50%) after expansion, i.e. x= (U/Y) -F.

CPU capacity expansion model: and (3) carrying out CPU pressure test before online, and obtaining the quantity of CPU resources required by the change of different data quantities according to the relation between the quantity of data produced and consumed by message middleware in the cluster and the CPU utilization rate, namely, a CPU model calculation formula, wherein if the quantity of data produced and consumed by the message middleware in the time periods of 1 minute, 15 minutes and 60 minutes is changed, an analysis module calculates the increment of the CPU utilization rate caused by the increment of the data quantity in the time periods and the estimated more than 80% of the CPU utilization rate of the existing cluster, and the analysis module can calculate the CPU capacity expansion quantity according to the increment of the data quantity of the message middleware.

The CPU model calculation formula: and X=1.5Y, wherein the total number of the consumption node machines before capacity expansion is Y, and the total number of the consumption node machines after capacity expansion is X=1.5Y, namely 50% of the consumption machines are increased by the machines before capacity expansion, and the goal is to reduce the number of the consumption machines to below 20%, so that the cluster calculation is ensured to be stable.

In the module, the change of the data volume of the Topic of the message middleware is required to be analyzed, the analysis module calculates the change condition of the production and consumption of the message middleware data to trigger the resource value of capacity expansion, and simultaneously records the information of the capacity expansion and sends the information to the issuing module to perform cluster capacity expansion.

The clusters in the module are generally referred to as cloud clusters, so parameters in the expansion model are adjusted according to the actual cloud cluster composition.

S400, carrying out cluster capacity expansion according to the capacity expansion information.

It should be noted that, the capacity expansion information includes disk capacity expansion information and/or machine capacity expansion information, and in some embodiments, step S400 may include: performing disk capacity expansion on the cluster machine according to the disk capacity expansion information; and/or configuring the new machine according to the machine capacity expansion information, adding the configured new machine into the cluster machine, and completing the capacity expansion of the cluster machine.

In some embodiments, the delivery module may receive the capacity expansion information and implement capacity expansion, and illustratively, the delivery module receives the capacity expansion information, performs network configuration and cluster software configuration update of the new machine, and delivers the network configuration file and the cluster software to the new machine. Meanwhile, the cluster software is restarted through the client agent, the client agent of the new machine receives the issuing instruction of the master control module, the client agent executes the network configuration information such as the updated new machine firewall list issued by the issuing module, then executes the cluster software configuration information issued by the issuing module, executes the software configuration of the updated new machine, restarts the software of the newly added machine, simultaneously informs the issuing module that the network and the cluster software configuration of the new machine are successfully updated, the issuing module updates the cluster configuration, and the new machine is added into the cluster.

In some embodiments, as shown in fig. 6, the method further includes: s500, acquiring a capacity expansion operation result of the cluster machine after completing the capacity expansion of the cluster; and S600, completing cluster capacity expansion when the capacity expansion operation result reaches the expected operation range, otherwise, performing cluster capacity expansion again based on the capacity expansion operation result.

It should be noted that, the capacity expansion operation result is consistent with the resource consumption data property of the foregoing steps, in some embodiments, the resource consumption target value may further include a CPU target usage rate, and when the CPU usage rate in the capacity expansion operation result does not conform to the CPU target usage rate, performing machine capacity expansion again; similarly, when the predicted data of the disk capacity consumption in the capacity expansion operation result does not accord with the target value of the disk capacity consumption, the disk capacity expansion is checked or directly carried out again. The principle of cluster capacity expansion again is identical to the principle of cluster capacity expansion in steps S300 and S400, and will not be described again.

In some embodiments, the capacity expansion result is further determined, and, for example, the client collects load, memory, disk usage information of the distributed cluster and running state information of the cluster software, and the analysis module determines whether the current cluster running is in the expected normal range, and if so, the capacity expansion is ended.

For the purpose of illustrating the principles of the present invention in detail, the following general flow chart of the present invention is described in connection with certain specific embodiments, and it is to be understood that the following is illustrative of the principles of the present invention and is not to be construed as limiting the present invention.

Firstly, it should be noted that, in general, the message middleware temporarily caches data, which belongs to a data source for a large data cluster, after the cache data of the message middleware is consumed, the data enters the large data cluster, and whether the large data cluster is to be expanded is judged by monitoring the load of the cluster, wherein the expansion is to be performed by the change of the load of the cluster, the comprehensive analysis of the change of the source data and the load capacity of the cluster is lacking, whether the cluster is to be expanded is pre-judged according to the comprehensive analysis, meanwhile, the reasonable operation interval of the load of the cluster can be reached only by how much storage and calculation resources which need to be expanded are not provided by the change of the source data, the expansion cost is high, and the expansion of the large data cluster is inflexible.

In view of this, the invention deploys the client agent acquisition component on all machines of the distributed cluster first, obtains the production and consumption capability information of all topics of the message middleware through the client agent program, collect the changes including about 1 minute, 15 minutes, 60 minutes of the partition of each Topic, collect the cluster information and report to the cluster logic analysis module, combine the analysis module to judge: according to the data quantity change of all the subjects of the current message middleware in a unit time period and the current cluster running indexes, comprehensively evaluating the current load value of the cluster due to the increase of the data quantity of the message middleware, judging whether the load value exceeds the pre-warning value born by the Hadoop, elasticSearch cluster, judging and recording the information exceeding the pre-warning value through analysis, including how much total disk space needs to be increased and how much CPU quantity needs to be expanded, informing a issuing module, completing the network configuration of a new machine and the dynamic configuration and issuing of cluster software, receiving the cluster software by a client agent of the new machine, and completing the restarting of the software. And simultaneously, the issuing assembly module issues instructions to all machines of the cluster, and the distributed cluster updates the information of the newly-added machines to complete the capacity expansion of the new machines. In some embodiments, as shown in fig. 7, the overall flow steps implemented in the method of the present invention are:

1. Client agent acquisition:

firstly, a agent system needs to be deployed on each machine, and the change of the data volume of the message middleware is mainly collected, wherein the change comprises information of cluster machine load, memory and disk use, and running state information of cluster software, such as information of production data volume of every 1 minute, 15 minutes, 60 minutes and the like of Partition on each Topic of all topics of the message middleware.

2. And (3) data collection and storage:

the client side reports the use information of each machine and the running state information of the software in the cluster to the database in real time through the interface, and the method is mainly used for analyzing the follow-up analysis modules.

3. And an analysis module:

the analysis module collects data quantity of message middleware production and consumption and machine resource consumption data, including CPU load, memory use, disk use and other operation parameters, and makes pressure test before online, obtains data of throughput, consumption, distributed cluster resource consumption and the like tested in each stage from the change of the data quantity of the message middleware and the distributed cluster resource consumption, obtains a maximum value and normal value data list which can be born by the message middleware production and consumption data quantity and corresponding clusters, generally considers the problems of redundancy, system space and the like, and also considers the space required by the disk in calculation, generally requires 30% of total disk capacity redundancy, so that the data quantity of the message middleware production and consumption is added with the current load value of the clusters, the predicted cluster disk capacity exceeds 70%, or the CPU utilization reaches more than 80%.

CPU capacity expansion model: and (3) carrying out CPU pressure test before online, and carrying out relation between the data quantity produced and consumed by message middleware and the CPU utilization rate in the cluster to obtain the quantity of CPU resources required after different data quantities are changed, namely a CPU model calculation formula, wherein if the data quantity of the message middleware in the time periods of 1 minute, 15 minutes and 60 minutes is changed, an analysis module calculates that the data quantity increase in the time periods and the CPU utilization rate of the existing cluster are estimated to be more than 80%, and an analysis module can calculate the CPU capacity expansion quantity according to the increase of the data quantity of the message middleware.

4. The issuing module receives the information and expands the capacity:

and the issuing module receives the capacity expansion information, respectively performs network configuration and cluster software configuration updating of the new machine, and issues a network configuration file and cluster software to the new machine.

5. The client agent restarts the cluster software:

the client agent of the new machine receives the issuing instruction of the master control module, executes the network configuration information such as the firewall list of the new machine, and the like, issued by the issuing module, then executes the cluster software configuration information issued by the issuing module, executes the software configuration of the updated machine, restarts the software of the newly added machine, simultaneously informs the issuing module that the network and the cluster software configuration of the new machine are successfully updated, and the issuing module updates the cluster configuration and adds the new machine into the cluster.

6. Judging the capacity expansion result:

the client collects load, memory, disk usage information and running state information of the distributed cluster, and an analysis module judges whether the current cluster runs in an expected normal range or not, and if the current cluster runs in the normal range, the capacity expansion is finished.

In summary, the invention analyzes the change of the data volume of the message middleware and the comprehensive resource use condition of the current cluster load, constructs the relation of logic judgment triggering capacity expansion condition values, obtains the judgment of whether the cluster is to be expanded, then carries out automatic capacity expansion of the cluster, and simultaneously obtains the resource volume required by the capacity expansion of the cloud cluster, wherein the resource volume comprises the storage quantity of the capacity expansion and the quantity of the CPU, thereby avoiding high capacity expansion cost and improving the capacity expansion efficiency of the distributed cloud cluster. It should be noted that, in the current industry, the message middleware is not the main judgment basis of the current capacity expansion, according to the invention, the data in the message middleware is required to flow into the distributed cluster, and firstly, a logic judgment analysis module is constructed according to the data quantity change of the flowing message middleware, so as to judge whether the data quantity change can cause the capacity expansion of the cluster. At present, the traditional cluster capacity expansion is to judge whether the load of the cluster exceeds a threshold value and then to carry out the cluster capacity expansion, but the invention obtains the resources needing capacity expansion by an analysis module through the change of the data volume of the message middleware, and can automatically issue software and start, thereby saving manpower. At present, the capacity expansion of the distributed cloud clusters is judged by the change of the data quantity of the production and consumption of the message middleware, which is still the first application in the industry, and the automatic capacity expansion of the clusters can be rapidly and timely carried out through the logic analysis of the production and consumption of the message middleware data, so that the high capacity expansion cost is avoided, and the capacity expansion efficiency of the distributed cloud clusters is improved;

Compared with the prior art, the main creativity is as follows:

1. at present, the traditional cluster capacity expansion is to determine whether to expand according to the cluster load, and no case of expanding based on the change of the data quantity of the message intermediate exists.

2. The invention provides an automatic capacity expansion method and device for a cloud cluster based on a message middleware, which are based on automatic analysis of all Topic related production and consumption capacities of the message middleware, and can analyze the change of the data volume of the message middleware data production and consumption by analyzing, and can predict whether to expand in advance and then perform automatic capacity expansion of the cluster, thereby avoiding low efficiency of manual capacity expansion and improving the capacity expansion efficiency of sub-clusters.

On the other hand, as shown in fig. 8, an embodiment of the present invention provides a cluster expansion device 800, including: a first module 810, configured to obtain data volume change information of the message middleware and resource consumption data of the cluster machine; a second module 820 for obtaining resource consumption prediction data according to the resource consumption data and the data quantity variation information; a third module 830, configured to determine expansion information based on the target value of resource consumption when the predicted data of resource consumption reaches a preset threshold; and a fourth module 840, configured to perform cluster capacity expansion according to the capacity expansion information.

In some embodiments, the apparatus may further include: a fifth module, configured to obtain a relational data list of the data amount change information and the newly added resource consumption data based on a stress test on the clustered machine, and determine a preset parameter; wherein the preset parameter includes at least one of a preset threshold value and a resource consumption target value.

In some embodiments, the apparatus may further include: a sixth module, configured to obtain a capacity expansion operation result of the clustered machine after completing the capacity expansion of the cluster; and a seventh module, configured to complete cluster capacity expansion when the capacity expansion operation result reaches the expected operation range, and if not, perform cluster capacity expansion again based on the capacity expansion operation result.

In some embodiments, the apparatus of the present invention is applied to a system architecture to implement cluster expansion, where the system architecture mainly includes: the client agent data acquisition and analysis module;

client agent data acquisition: the data is derived from the information of the data quantity change in the message intermediate and cluster load information, including information such as CPU, memory, disk and the like, and the acquired data is stored on Hdfs and mysql databases of the Hadoop cluster through data processing and data modeling, is subsequently processed through Spark MLlib and python programs, and is provided for an analysis module to use

And an analysis module: in the method, a pressure test is carried out immediately before the cluster is online, characteristic data such as throughput, consumption, distributed cluster resource consumption and the like of each stage of test are obtained from the change of the data quantity of the message middleware and the distributed cluster resource consumption, and the condition values of the corresponding distributed cluster triggering capacity expansion are obtained, for example, a cluster disk capacity expansion algorithm adopts U/(F+X) =Y, wherein the used value before capacity expansion is (U), the total value before capacity expansion is (F), the resource value (X) to be newly increased is the current time, the final load value of the total resource after capacity expansion is (Y), namely X= (U/Y) -F.

The content of the method embodiment of the invention is suitable for the device embodiment, the specific function of the device embodiment is the same as that of the method embodiment, and the achieved beneficial effects are the same as those of the method.

On the other hand, as shown in fig. 9, an embodiment of the present invention further provides an electronic device 900, which includes at least one processor 910, and at least one memory 920 for storing at least one program; take a processor 910 and a memory 920 as examples.

The processor 910 and the memory 920 may be connected by a bus or other means.

Memory 920 acts as a non-transitory computer readable storage medium that may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, memory 920 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some implementations, the memory 920 may optionally include memory located remotely from the processor, which may be connected to the device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The above described embodiments of the electronic device are merely illustrative, wherein the units described as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In particular, FIG. 10 schematically shows a block diagram of a computer system for implementing an electronic device of an embodiment of the invention.

It should be noted that, the computer system 1000 of the electronic device shown in fig. 10 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present invention.

As shown in fig. 10, the computer system 1000 includes a central processing unit 1001 (Central Processing Unit, CPU) which can execute various appropriate actions and processes according to a program stored in a Read-Only Memory 1002 (ROM) or a program loaded from a storage section 1008 into a random access Memory 1003 (Random Access Memory, RAM). In the random access memory 1003, various programs and data necessary for the system operation are also stored. The cpu 1001, the rom 1002, and the ram 1003 are connected to each other via a bus 1004. An Input/Output interface 1005 (i.e., an I/O interface) is also connected to bus 1004.

The following components are connected to the input/output interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output portion 1007 including a Cathode Ray Tube (CRT), a liquid crystal display (Liquid Crystal Display, LCD), and a speaker; a storage portion 1008 including a hard disk or the like; and a communication section 1009 including a network interface card such as a local area network card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The drive 1010 is also connected to the input/output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in the drive 1010, so that a computer program read out therefrom is installed as needed in the storage section 1008.

In particular, the processes described in the various method flowcharts may be implemented as computer software programs according to embodiments of the invention. For example, embodiments of the present invention include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1009, and/or installed from the removable medium 1011. The computer programs, when executed by the central processor 1001, perform the various functions defined in the system of the present invention.

It should be noted that, the computer readable medium shown in the embodiments of the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), flash Memory, an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The content of the method embodiment of the invention is suitable for the system embodiment, the specific function of the system embodiment is the same as that of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Another aspect of the embodiments of the present invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the foregoing method.

The content of the method embodiment of the invention is applicable to the computer readable storage medium embodiment, the functions of the computer readable storage medium embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the foregoing method.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It should be noted that although in the above detailed description several modules of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present invention.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution apparatus, device, or apparatus, such as a computer-based apparatus, processor-containing apparatus, or other apparatus that can fetch the instructions from the instruction execution apparatus, device, or apparatus and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution apparatus, device, or apparatus.

More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.

While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and the equivalent modifications or substitutions are intended to be included in the scope of the present invention as defined in the appended claims.

Claims

1. A cluster expansion method, comprising:

obtaining resource consumption prediction data according to the resource consumption data and the data quantity information;

when the resource consumption prediction data reaches a preset threshold value, determining capacity expansion information based on a resource consumption target value;

2. The cluster expansion method according to claim 1, wherein the obtaining the data volume change information of the message middleware and the resource consumption data of the cluster machine includes:

Acquiring the data quantity of production and consumption of all subjects of the message middleware in a preset time period as the data quantity information of the message middleware;

3. The cluster expansion method according to claim 1, further comprising:

based on the pressure test of the cluster machine, obtaining a relation data list of the data quantity change information and newly-added resource consumption data, and determining preset parameters; wherein the preset parameter includes at least one of the preset threshold value and the target value of resource consumption.

4. The cluster expansion method according to claim 1, wherein the obtaining resource consumption prediction data according to the resource consumption data and the data amount variation information includes:

and obtaining resource consumption prediction data according to the resource consumption data and the newly-added resource consumption data.

5. The cluster expansion method according to claim 1, wherein the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; when the predicted data of resource consumption reaches a preset threshold, determining capacity expansion information based on a target value of resource consumption includes:

when the predicted data of the disk capacity consumption reaches the first threshold value, acquiring the total space of the disk capacity, and subtracting the total space of the disk capacity from the ratio of the data of the disk capacity consumption to the target value of the disk capacity consumption to obtain the expansion information of the disk;

and/or, when the CPU utilization rate reaches the second threshold value, acquiring the number of machines of the cluster machine, and acquiring the machine capacity expansion information according to the number of machines and a preset proportion.

6. The cluster expansion method according to claim 1, wherein the expansion information includes disk expansion information and/or machine expansion information; the performing cluster capacity expansion according to the capacity expansion information includes:

and/or configuring a new machine according to the machine capacity expansion information, and adding the configured new machine into the cluster machine to complete the capacity expansion of the cluster machine.

7. The cluster expansion method according to claim 1, further comprising:

acquiring a capacity expansion operation result of the cluster machine after the cluster capacity expansion is completed;

8. A cluster expansion device, comprising:

the second module is used for obtaining resource consumption prediction data according to the resource consumption data and combining the data quantity change information;

a third module, configured to determine capacity expansion information based on a target value of resource consumption when the predicted data of resource consumption reaches a preset threshold;

9. An electronic device comprising a processor and a memory;

the memory is used for storing programs;

the processor executing the program implements the method of any one of claims 1 to 7.

10. A computer storage medium in which a processor executable program is stored, characterized in that the processor executable program is for implementing the method according to any one of claims 1 to 7 when being executed by the processor.