CN114490066A

CN114490066A - Method and device for dynamically adjusting large data cluster resources

Info

Publication number: CN114490066A
Application number: CN202210105773.0A
Authority: CN
Inventors: 顾凌云; 郭志攀; 王伟; 张爱平; 李军军; 陈波
Original assignee: Nanjing Bingjian Information Technology Co ltd
Current assignee: Nanjing Bingjian Information Technology Co ltd
Priority date: 2022-01-28
Filing date: 2022-01-28
Publication date: 2022-05-13

Abstract

The application provides a method and a device for dynamically adjusting big data cluster resources, wherein the embodiment automatically monitors that the big data cluster has a current capacity expansion requirement, and acquires the current available cloud server information of a public cloud; and sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and available cloud server information, and selecting at least one from corresponding sequencing results as a target cloud server to apply the target cloud server to perform resource capacity expansion on the big data cluster. The method and the device can realize dynamic adjustment of the big data cluster resources, and can effectively improve timeliness and automation degree of the dynamic adjustment of the big data cluster resources, thereby improving reliability and applicability of dynamic adjustment results of the resources.

Description

Method and device for dynamically adjusting large data cluster resources

Technical Field

The application relates to the technical field of computers, in particular to a method and a device for dynamically adjusting big data cluster resources.

Background

In the big data cluster, big data nodes are used for calculating big data tasks, and the capacity of rapidly processing massive big data sets is achieved through mobile calculation. Usually, a big data cluster is built before calculation, and a calculation task is sent to the cluster to process data at a later stage. According to the scheme, the approximate cluster resource use condition needs to be estimated during calculation and construction, and cluster machine nodes cannot be increased or reduced according to the cluster resource use condition at a later stage, for example, too few resource evaluations result in that too many cluster tasks lead to support tasks in a waiting state for a long time, too many cluster resource evaluations result in that clusters are in an idle state for a long time, and resource waste is caused. Therefore, the resources of the data cluster need to be adjusted.

At present, the existing big data cluster resource adjustment mode usually evaluates the resource usage condition manually, and then adds or reduces servers in the big data cluster, if the resource usage condition shows that capacity expansion is needed, a capacity expansion server is selected to join the big data cluster. However, the method of checking the resource use condition by using manpower irregularly has the problems of high error probability caused by a plurality of manual intervention steps, incapability of responding to the resource condition of the cluster in real time, possibility of causing delay to resource evaluation and the like; in the process of selecting the capacity expansion server, because no automatic and effective selection means exists, the problems of poor applicability and reliability of the resource adjustment result and the like are easily caused.

That is to say, the existing big data cluster resource adjusting mode has the problems of low automation degree of resource adjustment, untimely response, poor reliability, poor applicability and the like.

Disclosure of Invention

In order to overcome the above disadvantages in the prior art, an object of the present application is to provide a method and an apparatus for dynamically adjusting big data cluster resources, which can dynamically adjust big data cluster resources, and can effectively improve timeliness and automation degree of the dynamic adjustment of big data cluster resources, thereby improving reliability and applicability of a result of the dynamic adjustment of resources.

In order to solve the technical problem, the application provides the following technical scheme:

in a first aspect, the present application provides a method for dynamically adjusting big data cluster resources, including:

if the current capacity expansion requirement of the big data cluster is automatically monitored, the current available cloud server information of the public cloud is obtained;

sequencing currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information, and selecting at least one from corresponding sequencing results as a target cloud server;

and applying the target cloud server to perform resource capacity expansion on the big data cluster.

Further, still include:

if the current capacity reduction requirement of the big data cluster is automatically monitored, a current target cloud server list of the big data cluster is called from the public cloud;

and judging whether each target cloud server in the target cloud server list contains a current idle target cloud server, if so, releasing the current idle target cloud server in the big data cluster, and deleting the released target cloud server in the target cloud server list.

Further, before the step of obtaining information of a current available cloud server of a public cloud if the current capacity expansion requirement of the big data cluster is automatically monitored, the method further includes:

acquiring resource use data of the big data cluster in real time based on a preset monitoring program;

comparing the current resource usage data with a preset resource usage value range;

correspondingly, if it is automatically monitored that the large data cluster currently has a capacity expansion requirement, then the current available cloud server information of the public cloud is obtained, which includes:

and if the resource usage data is larger than the upper limit value of the resource usage numerical range, judging that the big data cluster currently has capacity expansion requirements, and acquiring the current available cloud server information of the public cloud.

Further, before the sorting the currently available servers in the public cloud, the method further includes:

acquiring capacity expansion attributes corresponding to the resource use data;

and generating the current capacity expansion rule of the big data cluster according to the capacity expansion attribute, wherein the capacity expansion attribute comprises at least one of quantity requirement data, performance requirement data, resource quantity requirement data and purchase price requirement data of available servers.

Further, the capacity expansion attribute comprises quantity requirement data and purchase price requirement data;

correspondingly, the sorting the currently available cloud servers in the public cloud based on the preset capacity expansion rule and the available cloud server information, and selecting at least one from the corresponding sorting results as a target cloud server, includes:

according to the capacity expansion rule corresponding to the quantity requirement data and the purchase price requirement data, sorting the currently available cloud servers in the public cloud from low prices to high prices to obtain a corresponding sorting result;

and selecting the first N cloud servers from the sequencing result to purchase and serve as target cloud servers, wherein N is determined according to the quantity requirement data, and is a positive integer equal to or larger than 1.

Further, before the collecting the resource usage data of the big data cluster in real time based on the preset monitoring program, the method further includes:

establishing a big data cluster, wherein the big data cluster comprises a master node server and a slave node server;

the main node server is used for carrying out task scheduling and resource management on the big data cluster;

and the slave node server is used for receiving the tasks distributed by the master node server and executing corresponding task calculation.

In a second aspect, the present application provides a big data cluster resource dynamic adjustment apparatus, including: the capacity expansion module is used for executing the following contents:

Further, still include: a capacity reduction module for executing the following contents:

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the big data cluster resource dynamic adjustment method when executing the computer program.

In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the big data cluster resource dynamic adjustment method.

Compared with the prior art, the method has the following beneficial effects:

by automatically monitoring the big data cluster, the automation degree of the big data cluster monitoring can be effectively improved, the response can be timely made when the current capacity expansion requirement of the big data cluster is monitored, the dynamic adjustment of big data cluster resources can be realized, and the timeliness of the dynamic adjustment of the big data cluster resources can be effectively improved; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

Fig. 1 is a general flowchart of a method for dynamically adjusting big data cluster resources in an embodiment of the present application;

fig. 2 is a schematic preferred flow chart of a method for dynamically adjusting big data cluster resources in the embodiment of the present application;

fig. 3 is a schematic structural diagram of a big data cluster resource dynamic adjustment apparatus in an embodiment of the present application;

fig. 4 is a schematic view of a dynamic capacity expansion monitoring process in an application example of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

In the description of the present application, it is noted that the terms "first", "second", "third", and the like are used merely for distinguishing between descriptions and are not intended to indicate or imply relative importance.

In one or more embodiments of the present application, big data (big data) or huge data refers to data that is too massive to be captured, managed, processed, and organized in a reasonable time by mainstream software tools to help enterprise business decisions more actively.

The big data cluster resource adjustment mode can add or reduce machine nodes by manually evaluating the resource use condition and purchasing the machine nodes through an SDK API provided by a public cloud, and the general steps of the method are as follows:

the method comprises the following steps of (1) manually checking the use condition of the big data cluster resources through a WEB interface at irregular intervals.

If the big data cluster resources are in a tension state for a long time, new evaluation is carried out to estimate the number of machine nodes needing to be added, after the machine nodes are purchased, NodeManager related software is installed and started to be added into the big data cluster, and the resources are increased.

And if the big data cluster resources are found to be in an idle state for a long time, new evaluation is carried out to estimate the number of the machine nodes needing to be reduced, and the machine nodes needing to be quitted are logged in to manually stop the NodeManager to quit the big data cluster, so that the resources are reduced.

And fourthly, the increase and the reduction of cluster resources are realized through the steps II and III, and finally, the reasonable utilization of the resources is realized.

However, in the method, the resource use condition is checked by using manpower irregularly, so that the manual intervention steps are more, and the error probability is high; and by using a manual monitoring mode, the resource condition of the cluster cannot be responded in real time, and the resource evaluation may be delayed.

Therefore, a need exists for a method for determining the use condition of cluster resources more quickly, and realizing dynamic increase and decrease of machine nodes. And the cost of the server is reasonably controlled under the condition of meeting the requirement of using resources by the task.

Based on the above, aiming at the problems of low automation degree of resource adjustment, untimely response, poor reliability, poor applicability and the like existing in the existing big data cluster resource adjustment mode, the application provides a big data cluster resource dynamic adjustment method, which can effectively improve the automation degree of big data cluster monitoring by automatically monitoring a big data cluster, can timely respond when the big data cluster is monitored to have capacity expansion requirements currently, can realize dynamic adjustment of big data cluster resources, and can effectively improve the timeliness of the big data cluster resource dynamic adjustment; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

In one or more embodiments of the present application, a public cloud generally refers to a cloud that can be used by a third-party provider for a user, the public cloud is generally available through the Internet, and may be free or low-cost, and a core attribute of the public cloud is a shared resource service. There are many instances of such a cloud that can provide services throughout the open public network today.

In one or more embodiments of the present application, the NodeManager manages each node in a YARN cluster. The NodeManager provides services for each node in the cluster, from overseeing lifetime management of a container to monitoring resources and tracking node health. MRv1 manage the execution of Map and Reduce tasks through slots, while NodeManager manages abstract containers that represent resources for each node that are available to a particular application. YARN continues to use the HDFS layer. Its main NameNode is used for metadata services and DataNode is used for replicated storage services scattered in a cluster.

Based on the above, the present application further provides a dynamic big data cluster resource adjusting device for implementing the dynamic big data cluster resource adjusting method provided in one or more embodiments of the present application, where the dynamic big data cluster resource adjusting device may be in communication connection with a master node server, a public cloud, and a client device held by a user in a big data cluster, by itself or through a third party server, and the like, and the dynamic big data cluster resource adjusting device may be an independent server, or may be functionally integrated locally in the master node server, and may be specifically set according to actual needs.

It is understood that the client devices may include smart phones, tablet electronic devices, network set-top boxes, portable computers, desktop computers, Personal Digital Assistants (PDAs), in-vehicle devices, smart wearable devices, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.

The server and the client device may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (Remote Procedure Call Protocol), a REST Protocol (Representational State Transfer Protocol), and the like used above the above Protocol.

The following embodiments and application examples are specifically and individually described in detail.

In order to solve the problems of low automation degree of resource adjustment, untimely response, poor reliability, poor applicability and the like in the existing big data cluster resource adjustment method, the application provides an embodiment of a big data cluster resource dynamic adjustment method, and referring to fig. 1, the big data cluster resource dynamic adjustment method executed by a big data cluster resource dynamic adjustment device specifically includes the following contents:

step 100: and if the current capacity expansion requirement of the big data cluster is automatically monitored, the current available cloud server information of the public cloud is obtained.

It can be understood that the currently available cloud server information of the public cloud refers to information of cloud servers which are currently idle and are openly purchased by the public cloud, and may be specifically embodied as a cloud server list.

In an example of step 100, prices of cloud servers provided by a public cloud can be accessed using an API interface provided by the public cloud and arranged in ascending order according to the prices to obtain a list of cloud servers that can be purchased.

Step 200: and sequencing currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information, and selecting at least one from corresponding sequencing results as a target cloud server.

It may be understood that the capacity expansion rule may be rule data that is pre-stored in the big data cluster resource and dynamically adjusts locally, may also be rule data that is received from a client device held by a user, and may also be rule data that is generated in a targeted manner according to the current resource usage data of the big data cluster and has high adaptability.

Step 300: and applying the target cloud server to perform resource capacity expansion on the big data cluster.

In step 300, the step of performing resource expansion on the big data cluster by using the target cloud server means that the target cloud server is purchased from the public cloud, and then the target cloud server is added into the big data cluster for task processing.

In an example of step 300, a cloud server may be purchased from a cloud server list from the lowest price, an SDK provided by a public cloud may be called to purchase and start the cloud server, and the cloud server may use a server image of a big data cluster packaged in advance. And then querying a successfully purchased cloud server by using the SDK provided by the public cloud. Then the following steps are carried out:

1. SSH logs in the cloud server by using a user name and a password;

2. downloading hosts, join _ cluster.sh and leave _ cluster.sh which are pre-stored in a slave node server S to the local by using a wget command;

3. the downloaded hosts are used to overlay the hosts files (/ etc/hosts) that the system originally existed.

4. And modifying the current host name and restarting the network card.

5. Sh script is executed, and the current machine is added into the big data cluster to participate in task calculation.

As can be seen from the above description, the method for dynamically adjusting big data cluster resources provided in the embodiment of the present application can effectively improve the automation degree of big data cluster monitoring by automatically monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity expansion requirement, so as to effectively improve the timeliness of the dynamic adjustment of big data cluster resources and implement the dynamic adjustment of big data cluster resources; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

In order to provide an effective means for selecting a capacity reduction server, referring to fig. 2, an embodiment of the method for dynamically adjusting big data cluster resources provided in the present application further includes the following steps:

step 400: if the current capacity reduction requirement of the big data cluster is automatically monitored, a current target cloud server list of the big data cluster is called from the public cloud;

step 500: when each target cloud server in the target cloud server list comprises a current idle target cloud server, releasing the current idle target cloud server in the big data cluster, and deleting the released target cloud server in the target cloud server list.

It can be appreciated that when cluster resources are found to be idle, a portion of the purchased cloud servers need to be released to reduce the number of machines and reduce the large data cluster resources.

In an example of

steps

400 and 500, the SDK provided by the public cloud is used to query the purchased cloud servers to obtain a list of purchased servers. And (3) performing parallel login on the server list, checking whether a process of a calculation task exists, and if the process does not exist, executing a pre-downloaded leave _ cluster. And releasing all the servers of the obtained list needing to be released by using the SDK provided by the public cloud.

As can be seen from the above description, the method for dynamically adjusting the big data cluster resources provided in the embodiment of the present application can effectively improve the automation degree of monitoring the big data cluster by automatically monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity reduction demand, thereby effectively improving the timeliness of dynamically adjusting the big data cluster resources and realizing dynamic adjustment of the big data cluster resources; by judging whether each target cloud server in the target cloud server list contains a current idle target cloud server or not, if yes, releasing the current idle target cloud server in the big data cluster, and deleting the released target cloud server in the target cloud server list, an effective means for selecting a capacity reduction server can be provided, so that the automation degree and the intelligent degree of the capacity reduction process of the big data cluster can be effectively improved, and the reliability and the applicability of a capacity reduction result can be improved.

In order to further improve the timeliness and reliability of the dynamic adjustment of the big data cluster resource, in an embodiment of the dynamic adjustment method for the big data cluster resource provided by the present application, referring to fig. 2, before step 100 in the dynamic adjustment method for the big data cluster resource, the following contents are further included:

step 010: acquiring resource use data of the big data cluster in real time based on a preset monitoring program;

step 020: and comparing the current resource use data with a preset resource use numerical range, and judging whether the current resource use data is larger than the upper limit value or smaller than the lower limit value of the preset resource use numerical range.

Correspondingly, referring to fig. 2, step 100 in the method for dynamically adjusting big data cluster resources specifically includes the following contents:

step 110: and if the resource usage data is larger than the upper limit value of the resource usage numerical range, judging that the big data cluster currently has capacity expansion requirements, and acquiring the current available cloud server information of the public cloud.

It is understood that the specific implementation of step 400 may also be included:

step 410: if the resource usage data is smaller than the lower limit value of the resource usage numerical range, judging that the big data cluster has a capacity reduction requirement at present, and calling a current target cloud server list of the big data cluster from the public cloud.

And if the resource usage data is within the resource usage numerical range, judging that the current resource usage condition of the big data cluster is normal without carrying out capacity expansion or capacity reduction treatment.

As can be seen from the above description, the method for dynamically adjusting the big data cluster resources provided in the embodiment of the present application can more quickly determine the use condition of the big data cluster resources by acquiring the resource use data of the big data cluster in real time based on a preset monitoring program, so as to dynamically increase and decrease machine nodes; by comparing the current resource usage data with a preset resource usage value range, the timeliness and reliability of the dynamic adjustment of the big data cluster resources can be further improved.

In order to further improve the effectiveness, applicability, and intelligent degree of selecting a capacity expansion server, in an embodiment of the method for dynamically adjusting big data cluster resources provided in the present application, referring to fig. 2, before step 100 and step 200, the method for dynamically adjusting big data cluster resources further includes the following steps:

step 030: and acquiring the capacity expansion attribute corresponding to the resource use data.

Step 040: and generating the current capacity expansion rule of the big data cluster according to the capacity expansion attribute, wherein the capacity expansion attribute comprises at least one of quantity requirement data, performance requirement data, resource quantity requirement data and purchase price requirement data of available servers.

As can be seen from the above description, in the method for dynamically adjusting big data cluster resources provided in the embodiment of the present application, the current capacity expansion rule of the big data cluster is generated according to the capacity expansion attribute corresponding to the resource usage data, so that the pertinence and applicability of the capacity expansion rule to the resource usage data of the current big data cluster can be effectively improved, and further, the application reliability and applicability of the target cloud server selected according to the capacity expansion rule can be effectively improved; the capacity expansion attribute comprises at least one of quantity requirement data, performance requirement data, resource quantity requirement data and purchase price requirement data of the available servers, and the pertinence and the applicability of the generated current capacity expansion rule of the big data cluster can be further improved.

In order to reasonably control the cost of the server under the condition of meeting the task resource usage requirement, in an embodiment of the big data cluster resource dynamic adjustment method provided by the application, the capacity expansion attribute comprises quantity requirement data and purchase price requirement data; referring to fig. 2, step 200 of the method for dynamically adjusting big data cluster resources specifically includes the following contents:

step 210: and sorting the currently available cloud servers in the public cloud from low prices to high prices according to the capacity expansion rule corresponding to the quantity requirement data and the purchase price requirement data to obtain a corresponding sorting result.

Step 220: and selecting the first N cloud servers from the sequencing result to purchase and serve as target cloud servers, wherein N is determined according to the quantity requirement data and is a positive integer equal to or larger than 1.

As can be seen from the above description, according to the dynamic adjustment method for big data cluster resources provided in this embodiment of the application, the currently available cloud servers in the public cloud are ranked from low price to high price according to the capacity expansion rule corresponding to the quantity requirement data and the purchase price requirement data, so as to obtain a corresponding ranking result, and when cluster resources are in shortage, a new server of a bidding instance of the public cloud can be immediately purchased and added to the big data cluster, so as to increase the cluster resources; the cost of the server can be reasonably controlled under the condition of meeting the requirement of using resources by the task.

In order to ensure the normal operation of the big data cluster on the basis of effectively avoiding resource waste, in an embodiment of the method for dynamically adjusting big data cluster resources provided in the present application, referring to fig. 2, the following contents are further specifically included before step 010 of the method for dynamically adjusting big data cluster resources:

step 001: and establishing a big data cluster, wherein the big data cluster comprises a master node server and a slave node server, the master node server is used for carrying out task scheduling and resource management of the big data cluster, and the slave node server is used for receiving the tasks distributed by the master node server and executing corresponding task calculation.

For example, a big data cluster is established, 2 machines are fixed, one cluster master node a is installed, and one slave node B is installed, and the big data cluster is mainly used as a computing node to exist in the cluster for a long time. Preventing that tasks can still be scheduled if a new machine fails to join the cluster. Mainly, ResourceManager (master node a) and NodeManager (slave node B) are installed.

ResourceManager: cluster task scheduling and resource management.

NodeManager: and receiving the tasks distributed by the ResourceMenager and executing task calculation.

As can be seen from the foregoing description, the method for dynamically adjusting resources of a big data cluster according to the embodiment of the present application establishes an initial cluster including a master node server and a slave node server, so that the slave node server is mainly used as a computing node and exists in the cluster for a long time.

In terms of software, in order to solve the problems of low automation degree, untimely response, poor reliability, poor applicability, and the like of resource adjustment in the existing big data cluster resource adjustment method, the present application provides an embodiment of a big data cluster resource dynamic adjustment apparatus for executing all or part of the contents in the big data cluster resource dynamic adjustment method, and referring to fig. 3, the big data cluster resource dynamic adjustment apparatus specifically includes the following contents:

a capacity expansion module 10, configured to execute the following:

sequencing currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information, and selecting at least one from corresponding sequencing results to serve as a target cloud server;

As can be seen from the above description, the dynamic big data cluster resource adjusting device provided in the embodiment of the present application can effectively improve the automation degree of big data cluster monitoring by automatically monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity expansion requirement, so as to effectively improve the timeliness of dynamic big data cluster resource adjustment, and implement dynamic adjustment of the big data cluster resource; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

In order to provide an effective means for selecting a capacity reduction server, in an embodiment of the apparatus for dynamically adjusting big data cluster resources provided in the present application, referring to fig. 3, the apparatus for dynamically adjusting big data cluster resources further includes the following contents:

a capacity reduction module 20, configured to perform the following:

The embodiment of the dynamic big data cluster resource adjustment apparatus provided in the present application may be specifically configured to execute the processing procedure of the embodiment of the dynamic big data cluster resource adjustment method in the foregoing embodiment, and the function of the processing procedure is not described herein again, and reference may be made to the detailed description of the embodiment of the method.

As can be seen from the above description, the dynamic big data cluster resource adjusting device provided in the embodiment of the present application can effectively improve the automation degree of big data cluster monitoring by automatically monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity reduction demand, so as to effectively improve the timeliness of dynamic big data cluster resource adjustment, and implement dynamic adjustment of the big data cluster resource; by judging whether each target cloud server in the target cloud server list contains a current idle target cloud server, if so, releasing the current idle target cloud server in the big data cluster, and deleting the released target cloud server in the target cloud server list, an effective means for selecting a capacity reduction server can be provided, so that the automation degree and the intelligent degree of the capacity reduction process of the big data cluster can be effectively improved, and the reliability and the applicability of a capacity reduction result can be improved.

For further explanation, the present application further provides a specific application example of dynamic adjustment of large data cluster resources, which is specifically embodied as a dynamic capacity expansion method for a large data cluster, and referring to fig. 4, the technical field to which the application example of the present application belongs is computer technology, and the application example aims to monitor the use condition of cluster resources to dynamically increase or decrease nodes so as to meet the requirement of suitable resources of a cluster.

The dynamic capacity expansion method for the big data cluster provided by the application example of the application example is mainly characterized in that when cluster resources are in shortage, a new server of a public cloud bidding example is immediately purchased and added to the big data cluster, and the cluster resources are increased; when the cluster resources are used too little, the public cloud bidding instance server is released to quit the big data cluster, and the cluster resources are reduced.

The method for dynamically expanding the big data cluster provided by the specific application example of the application specifically comprises the following contents:

step 1: and designating a machine as a server, and installing httpd, nginx, wget and other software.

It will be appreciated that httpd is the main program of the Apache hypertext transfer protocol (HTTP) server. Designed as an independently running background process, it establishes a pool of sub-processes or threads that handle requests.

It will be appreciated that nginx is a high performance HTTP and reverse proxy web server, while also providing IMAP/POP3/SMTP services. Nginx is a lightweight Web server/reverse proxy server and email (IMAP/POP3) proxy server, published under the BSD-like protocol. Its advantages are less memory occupied and high concurrency.

It will be appreciated that wget is a free tool for automatically downloading files from a network, supports downloading via the three most common TCP/IP protocols HTTP, HTTPS, FTP, and may use HTTP proxy. The name "wget" is derived from the combination of "World Wide Web" and "get".

Step 2: all switches used for establishing the big data cluster ensure that the network of the switches can be communicated with the big data cluster. And creating hosts, join _ cluster.sh and leave _ cluster.sh files, and storing the files in a corresponding directory of server software installed on the server S.

It is understood that a Switch means a "Switch" is a network device for electrical (optical) signal forwarding. It may provide an exclusive electrical signal path for any two network nodes accessing the switch. The most common switch is an ethernet switch. Other common are telephone voice switches, fiber switches, and the like.

hosts: all IP addresses supported by all switches, all server IP addresses in the big data cluster, one IP address on a row.

Mesh _ cluster.sh: and adding a new machine into the script of the cluster, starting to receive the task and participating in the calculation.

leave _ cluster.sh: and (4) exiting the current machine from the big data cluster to the script, and not accepting the task to participate in the calculation.

And step 3: the method comprises the steps of establishing a big data cluster, fixing 2 machines, installing a cluster main node A and installing a slave node B, and mainly using the big data cluster as a computing node to exist in the cluster for a long time. Preventing that tasks can still be scheduled if a new machine fails to join the cluster. Mainly, ResourceManager (master node a) and NodeManager (slave node B) are installed.

The basic idea of YARN is to separate two main functions (resource management and job scheduling/monitoring) of JobTracker, and the main method is to create a global resource manager (rm) and several application masters (am) for application programs. An application here refers to a traditional MapReduce job or DAG (directed acyclic graph) of a job.

ResourceManager: cluster task scheduling and resource management.

And 4, step 4: after the relevant big data components are installed on the slave node B, the system is packaged into a server image C.

It is understood that mirroring is a mirror server. The mirror image server and the main server have the same service content, and are only placed in a different place to share the load capacity of the main server. But not master may be used. Two or more servers, other than the primary server, on the web that have identical content and are updated synchronously are referred to as mirror servers.

And 5: and accessing the prices of the ECS cloud servers provided by the public cloud by using an API (application programming interface) provided by the public cloud, and arranging the prices in an ascending order to obtain a list E of the ECS cloud servers which can be purchased.

And 6: and purchasing the ECS cloud server from the ECS cloud server list E from the lowest price. And calling the SDK provided by the public cloud for purchase and starting an ECS cloud server, wherein the ECS uses the image C packaged in the step 4.

It can be understood that an ECS cloud server (electronic computer Service) is a simple, efficient, secure, reliable, and processing power scalable computing Service. The management mode is simpler and more efficient than that of a physical server. A user can rapidly create or release any plurality of cloud servers without purchasing hardware in advance.

And 7: step 6 is queried using the SDK provided by the public cloud to purchase a successful ECS cloud server.

Then the following steps are carried out:

1. SSH logs in to the ECS cloud server using a username and password.

It is understood that SSH is an abbreviation for Secure Shell, as established by the Network Group of IETF (Network Working Group); SSH is a security protocol built on an application layer basis. SSH is a relatively reliable protocol that is dedicated to providing security for telnet sessions and other web services. The SSH protocol can effectively prevent the problem of information leakage in the remote management process. SSH was initially a program on UNIX systems and was subsequently rapidly expanding to other operating platforms. SSH, when used correctly, can remedy vulnerabilities in the network. SSH clients are applicable to a variety of platforms. Almost all UNIX platforms-including HP-UX, Linux, AIX, Solaris, Digital UNIX, Irix, and others-can run SSH.

2. Downloading hosts, join _ cluster.sh and leave _ cluster.sh stored in the server S in the step 2 to the local by using a wget command;

4. And modifying the current host name and restarting the network card.

And 8: monitoring the use condition of the cluster resources by the monitor program through a Restful API of a resource manager, and executing the steps 5 to 7 when the cluster resources are found to be insufficient, and purchasing and adding a new machine; and when the cluster resources are found to be idle, executing the steps 9 to 11, releasing a part of the purchased ECS cloud servers to reduce the number of machines and reduce the large data cluster resources.

It is understood that restul is a design style and development approach for web applications, and based on HTTP, XML format definition or JSON format definition can be used. RESTFUL is suitable for a scene that a mobile internet manufacturer serves as a service interface, the function that a third party OTT calls mobile network resources is achieved, and the action type is to add, change and delete the called resources.

And step 9: and querying the purchased ECS cloud servers by using the SDK provided by the public cloud to obtain a purchased server list F.

Step 10: and (4) performing parallel login on the server list F, checking whether a process of a computing task exists, and if not, executing the leave _ cluster.

Step 11: and releasing all the servers of the ECS list D needing to be released, which are obtained in the step 10, by using the SDK provided by the public cloud.

From the above content, the application example of the application achieves increasing or reducing of big data cluster resources by monitoring the use condition of the big data cluster resources in real time and combining a public cloud SDK API mode to purchase and release the servers in real time. The method can ensure that the server resources are reasonably controlled under the condition of using the big data task resources to the maximum extent, ensures that resource waste and resource loss cannot be caused, and at least has the following beneficial effects:

firstly, when the server is added in the application example, the purchased server belongs to a public cloud and can meet task computing requirements, and meanwhile, the server with the lowest price reduces cost consumption. Meanwhile, the method can ensure that the resources required by the big data computing task can be quickly responded, and the big data computing task cannot be blocked for a long time due to lack of resources.

Secondly, the resource use condition of the big data cluster is monitored in real time through a monitoring program, once the fact that the server can be released to reduce the resources of the big data cluster is found, the node manager is automatically logged on the machine node of the big data cluster to stop the node manager, no task is guaranteed to run on the server to be stopped, and the situation that the server is mistakenly killed is reduced.

In terms of hardware, in order to solve the problems of low automation degree, untimely response, poor reliability, poor applicability, and the like of resource adjustment in the existing big data cluster resource adjustment method, the present application provides an embodiment of an electronic device for implementing all or part of the contents in the big data cluster resource dynamic adjustment method, where the electronic device specifically includes the following contents:

the electronic device may include a central processor and a memory; the memory is coupled to the central processor. Notably; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.

In one embodiment, the big data cluster resource dynamic adjustment function may be integrated into a central processor. Wherein the central processor may be configured to control:

Step 300: and carrying out resource expansion on the big data cluster by applying the target cloud server.

As can be seen from the above description, the electronic device provided in the embodiment of the present application, by automatically monitoring the big data cluster, can effectively improve the automation degree of monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity expansion requirement, so as to effectively improve the timeliness of dynamically adjusting the big data cluster resources, and implement dynamic adjustment of the big data cluster resources; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

In another embodiment, the big data cluster resource dynamic adjustment apparatus may be configured separately from the central processing unit, for example, the big data cluster resource dynamic adjustment apparatus may be configured as a chip connected to the central processing unit, and the big data cluster resource dynamic adjustment function is realized through the control of the central processing unit.

The electronic device may further include: communication module, input unit, audio processor, display, power. A central processing unit, also sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device that receives input and controls the operation of various components of the electronic device.

The memory may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit may execute the program stored in the memory to realize information storage or processing, or the like.

The input unit provides input to the central processing unit. The input unit is, for example, a key or a touch input device. The power supply is used to provide power to the electronic device. The display is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.

The memory may be a solid state memory, such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory may also be some other type of device. The memory includes a buffer memory 9141 (sometimes referred to as a buffer). The memory may include an application/function storage section for storing an application program and a function program or a flow for executing an operation of the electronic device by the central processor.

The memory may also include a data store for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion of the memory may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, directory applications, etc.).

The communication module is a transmitter/receiver that transmits and receives signals via an antenna. The communication module (transmitter/receiver) is coupled to the central processor to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be disposed in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone to implement the usual telecommunication functions. The audio processor may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor is also coupled to the central processor, so that recording on the local machine can be realized through the microphone, and sound stored on the local machine can be played through the loudspeaker.

An embodiment of the present application further provides a computer-readable storage medium capable of implementing all the steps in the dynamic big data cluster resource adjustment method in the foregoing embodiment, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the dynamic big data cluster resource adjustment method whose execution subject is a server or a client in the foregoing embodiment, or all the steps.

For example, the processor, when executing the computer program, implements the steps of:

As can be seen from the above description, the computer-readable storage medium provided in this embodiment of the present application, through automatically monitoring the big data cluster, can effectively improve the automation degree of monitoring the big data cluster, and can respond in time when it is monitored that the big data cluster currently has a capacity expansion requirement, so as to effectively improve the timeliness of dynamically adjusting the big data cluster resources, and implement dynamic adjustment of the big data cluster resources; by sequencing all currently available cloud servers in the public cloud based on preset capacity expansion rules and the available cloud server information and selecting at least one from corresponding sequencing results as a target cloud server, an effective means for selecting capacity expansion servers can be provided, the automation degree of the capacity expansion process of the big data cluster can be effectively improved, and the reliability and the applicability of the capacity expansion results can be improved.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above description is only for various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and all such changes or substitutions are included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A big data cluster resource dynamic adjustment method is characterized by comprising the following steps:

2. The big data cluster resource dynamic adjustment method according to claim 1, further comprising:

3. The method according to claim 1, wherein before the step of obtaining information about currently available cloud servers of a public cloud if it is automatically monitored that there is a current capacity expansion requirement for the big data cluster, the method further comprises:

4. The big data cluster resource dynamic adjustment method according to claim 3, further comprising, before the sorting currently available servers in the public cloud:

acquiring capacity expansion attributes corresponding to the resource use data;

5. The big data cluster resource dynamic adjustment method according to claim 4, wherein the capacity expansion attribute comprises quantity requirement data and purchase price requirement data;

and selecting the first N cloud servers from the sequencing result to purchase and serve as target cloud servers, wherein N is determined according to the quantity requirement data and is a positive integer equal to or larger than 1.

6. The big data cluster resource dynamic adjustment method according to claim 3, further comprising, before the collecting the resource usage data of the big data cluster in real time based on the preset monitoring program:

7. A big data cluster resource dynamic adjustment device is characterized by comprising: the capacity expansion module is used for executing the following contents:

8. The big data cluster resource dynamic adjustment apparatus according to claim 7, further comprising: a capacity reduction module for executing the following contents:

if the capacity reduction requirement of the big data cluster currently exists is automatically monitored, a current target cloud server list of the big data cluster is called from the public cloud;

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the big data cluster resource dynamic adjustment method of any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, the computer program, when being executed by a processor, implementing the big data cluster resource dynamic adjustment method according to any one of claims 1 to 6.