CN111782147B - Method and device for cluster expansion and contraction capacity - Google Patents
Method and device for cluster expansion and contraction capacity Download PDFInfo
- Publication number
- CN111782147B CN111782147B CN202010614916.1A CN202010614916A CN111782147B CN 111782147 B CN111782147 B CN 111782147B CN 202010614916 A CN202010614916 A CN 202010614916A CN 111782147 B CN111782147 B CN 111782147B
- Authority
- CN
- China
- Prior art keywords
- expansion
- contraction
- cluster
- conditions
- condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008602 contraction Effects 0.000 title claims abstract description 163
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000004044 response Effects 0.000 claims abstract description 16
- 230000015654 memory Effects 0.000 claims description 23
- 238000001514 detection method Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 2
- PXXLQQDIFVPNMP-UHFFFAOYSA-N 3-(diethylcarbamoyl)benzoic acid Chemical compound CCN(CC)C(=O)C1=CC=CC(C(O)=O)=C1 PXXLQQDIFVPNMP-UHFFFAOYSA-N 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application discloses a method and a device for expanding and shrinking capacity of a cluster, which can be used in the technical field of cloud computing. The specific implementation scheme is as follows: obtaining expansion and contraction conditions associated with a target cluster; in response to detecting that the capacity expansion and contraction conditions are met, performing capacity expansion and contraction operations corresponding to the capacity expansion and contraction conditions; determining the service type provided by the target cluster; and configuring the target cluster after the expansion and contraction operation is executed according to the service type. This embodiment improves the efficiency of cluster operation.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to the technical field of cloud computing.
Background
As the amount of data generated during the operation of enterprises increases, many enterprises choose to use cloud clusters for data processing. When the data processing operation is more, the cluster load is high, or when the data processing operation is less, the cluster load is low, the expansion and contraction are needed to improve the running efficiency of the cluster.
At present, expansion and contraction of the cloud cluster are realized mainly by manually increasing and decreasing resources in the cluster by a user.
Disclosure of Invention
The embodiment of the application provides a method, a device, equipment and a storage medium for cluster capacity expansion and contraction.
In a first aspect, some embodiments of the present application provide a method for cluster scaling, the method comprising: obtaining expansion and contraction conditions associated with a target cluster; in response to detecting that the capacity expansion and contraction conditions are met, performing capacity expansion and contraction operations corresponding to the capacity expansion and contraction conditions; determining the service type provided by the target cluster; and configuring the target cluster after the expansion and contraction operation is executed according to the service type.
In a second aspect, some embodiments of the present application provide an apparatus for cluster scaling, the apparatus comprising: an acquisition unit configured to acquire a capacity expansion condition associated with a target cluster; a capacity expansion and contraction unit configured to perform capacity expansion and contraction operations corresponding to capacity expansion and contraction conditions in response to detection that the capacity expansion and contraction conditions are satisfied; a first determining unit configured to determine a service type provided by the target cluster; and the configuration unit is configured to configure the target cluster after the expansion and contraction operation is performed according to the service type.
In a third aspect, some embodiments of the present application provide an apparatus comprising: one or more processors; and a storage device having one or more programs stored thereon, which when executed by the one or more processors cause the one or more processors to implement the method as described in the first aspect.
In a fourth aspect, some embodiments of the application provide a computer readable medium having stored thereon a computer program which when executed by a processor implements a method as described in the first aspect.
According to the technology provided by the application, the cluster operation efficiency is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are included to provide a better understanding of the present application and are not to be construed as limiting the application. Wherein:
FIG. 1 is a diagram of some exemplary system architecture in which the present application may be used;
FIG. 2 is a schematic diagram of a first embodiment according to the present application;
FIG. 3 is a schematic diagram of an application scenario according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a second embodiment according to the present application;
FIG. 5 is a schematic diagram of a third embodiment according to the present application;
FIG. 6 is a schematic diagram of a fourth embodiment according to the present application;
Fig. 7 is a schematic structural diagram of an electronic device suitable for implementing a method for cluster expansion and contraction according to an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of a method for cluster scaling or an apparatus for cluster scaling of the present application may be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various client applications, such as a cloud computing class application, a cloud service class application, a search class application, may be installed on the terminal devices 101, 102, 103. The terminal devices 101, 102, 103 may create a data processing job to be executed by the cluster.
The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices with display screens, including but not limited to smartphones, tablets, laptop and desktop computers, and the like. When the terminal devices 101, 102, 103 are software, they can be installed in the above-listed electronic devices. Which may be implemented as a plurality of software or software modules, or as a single software or software module. The present invention is not particularly limited herein.
The server 105 may be a server that provides various services, such as a server that manages a cluster. A cluster may include a plurality of physical computers, on which a plurality of Virtual Machines (VMs) may be installed, a complete computer system that is emulated by software and that has complete hardware system functionality and operates in a completely isolated environment.
The server 105 may obtain the expansion and contraction conditions associated with the target cluster; in response to detecting that the capacity expansion and contraction conditions are met, performing capacity expansion and contraction operations corresponding to the capacity expansion and contraction conditions; determining the service type provided by the target cluster; and configuring the target cluster after the expansion and contraction operation is executed according to the service type.
It should be noted that, the method for scaling a cluster according to the embodiment of the present application may be performed by the server 105, and accordingly, the device for scaling a cluster may be disposed in the server 105.
The server may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for cluster scaling is shown in accordance with the present application. The method for cluster expansion and contraction capacity comprises the following steps:
step 201, obtaining the expansion and contraction conditions associated with the target cluster.
In this embodiment, a method execution body (e.g., a server shown in fig. 1) for cluster scaling may obtain a scaling condition associated with a target cluster. The target cluster may be a user cluster for starting an automatic capacity expansion function in the user clusters managed by the execution body, and in addition, the execution body may also create, monitor and release a cluster.
Here, the expansion and contraction conditions may include an expansion condition and/or a contraction condition. The expansion and contraction conditions can be set on the basis of time or on the basis of indexes. The business scale has a time period rule, obvious peaks and troughs exist, for example, a daily report, a weekly report and other processing analysis scenes of specific time, the expansion and contraction conditions can be set based on time, the business change does not have a time rule, but the timely operation of important operations is required to be ensured, the cluster scale is required to be dynamically adjusted according to cluster load indexes, and the expansion and contraction conditions can be set based on the indexes.
The time-based set of expansion and contraction conditions may include a periodic expansion and contraction condition, for example, 8 a.m. each monday, and may also include an expansion and contraction condition at a specified time point, for example, 8 a.m. 20 a.m. 5 a.2020.
The scaling condition may be established based on an indicator in the monitoring information generated by the monitoring service of the cluster, e.g. the scaling condition is met when the indicator is greater or less than a preset threshold. In addition, statistical manners of cluster indexes, such as average value, maximum value, minimum value and the like, can be set. The monitoring information may include the load of the cluster, such as the memory usage of the resource manager, the cluster CPU load, etc. The monitoring information may also include resource parameters and/or the status of the node at which the resource is located. The resources may include at least one of: virtual machines, persistent databases (e.g., mySQL), cached databases (e.g., redis). The resource parameters may include CPU usage, memory usage, number of host connections, database usage, etc. The status of the node may include both surviving and non-surviving, the status of the node being detectable by means of heartbeat detection.
And step 202, in response to detecting that the capacity expansion and contraction conditions are met, executing capacity expansion and contraction operation corresponding to the capacity expansion and contraction conditions.
In this embodiment, the executing body may execute the expansion and contraction operation corresponding to the expansion and contraction condition in response to detecting that the expansion and contraction condition acquired in step 201 is satisfied. The expansion and contraction operation corresponds to expansion and contraction conditions, for example, the expansion and contraction conditions are that the memory utilization rate of the resource manager is between 70% and 80%, and the expansion and contraction operation corresponding to the expansion and contraction conditions can be to add 2 virtual machines. For another example, from 8 to 9 in the morning each day, if the cluster workload is large, the capacity expansion and contraction condition may be set to be 8 in the morning at the current time, the corresponding capacity expansion and contraction operation may be to increase a plurality of task nodes, the capacity expansion and contraction condition may be set to be 9 in the morning at the current time, and the corresponding capacity expansion and contraction operation may be to decrease a plurality of task nodes. In addition, the executing body responds to the fact that the expansion and contraction conditions are met, and can output transaction information of expansion and contraction transaction according to expansion and contraction operations corresponding to the expansion and contraction conditions; and executing the expansion and contraction operation corresponding to the expansion and contraction conditions in response to receiving the indication information representing the success of the expansion and contraction transaction.
The capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition can be set by a user, or can be determined based on historical data, for example, when the memory usage rate of the resource manager is higher than 80%, the memory usage rate of the resource manager can be reduced to a reasonable level by adding 2 virtual machines according to the historical data, and when the capacity expansion and contraction condition is set to be that the memory usage rate of the resource manager is higher than 80%, the capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition is added by 2 virtual machines.
In some optional implementations of this embodiment, the number of the scaling conditions associated with the target cluster may include one or more, and when the scaling conditions are plural, the priority of the scaling condition detection may be determined. In addition, a period for detecting the capacity expansion and contraction condition can be set, if the capacity expansion and contraction condition is not detected, the next detection can be waited for, if the capacity expansion and contraction condition is detected, whether the capacity expansion and contraction operation to be executed is still in the cooling time or not can be judged further, if the capacity expansion and contraction operation is executed, whether the number of resources in the cluster exceeds the preset number range can be judged, and if the capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition is not executed, the capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition can be executed.
In step 203, the service type provided by the target cluster is determined.
In this embodiment, the executing entity may determine the service type provided by the target cluster. The service type may characterize the service provided by the target cluster, which may be determined according to its functionality, and may include storage types such as data stores, operation types such as log processing, hybrid types, and so forth. The service type provided by the target cluster may also be determined according to the tool carried by the target cluster, for example, the carried tool is a distributed array-oriented open source database (Hbase), the service type of the target cluster may be determined to be the Hbase type, and the carried tool may further include a programming model (MapReduce) for parallel operation of a large-scale data set, a resource manager (yacn) of a distributed system infrastructure, a data warehouse tool (hive), a distributed file system (HDFS), and the like.
And 204, configuring the target cluster after the capacity expansion and contraction operation is executed according to the service type.
In this embodiment, the executing body may configure the target cluster after performing the expansion and contraction operation according to the service type determined in step 203. The configuration of the target cluster may include configuration of an environment, configuration of components, etc., parameters related to the configuration of the environment may be preset, and components in the configuration of the components may be pre-written. For example, the service type provided by the target cluster is a storage type, and a preset environment suitable for providing storage service can be configured for the service type, or a component providing storage service can be configured for the service type. For another example, if the target cluster is of Hbase type, a component responsible for storing actual data and a component for reading and writing data in response to a user request may be configured for the target cluster. In addition, the execution body may only configure the resource added by the expansion and contraction operation, or may configure other resources in the target cluster, for example, may reinstall or start a component for other resources in the target cluster according to the service type.
In some optional implementations of this embodiment, configuring the target cluster after performing the scaling operation according to the service type may include: the added resources for the scaling operation in the target cluster are configured with components matching the service type. For example, a Hbase, mapReduce, YARN-populated cluster, after performing the scaling operation, components such as regional services (regionServer), node management (nodemanager) and the like required for Hbase, mapReduce, YARN services may be automatically started for added resources, such as added virtual machines. In addition, the execution body can also establish a directed acyclic graph according to the dependency relationship among the components to be configured; determining the parallel configuration sequence of each component according to the directed acyclic graph; and configuring each component for the nodes according to the parallel configuration sequence so as to further improve the configuration efficiency. In the implementation manner, only the components matched with the service types are allocated for the added resources, so that more accurate allocation is realized, and the allocation efficiency is improved.
In the flow 200 of the method for expanding and shrinking the cluster in this embodiment, automatic expansion and shrinking of the target cluster are realized, and the target cluster after the expansion and shrinking operation is performed is configured according to the service type of the target cluster, so that the problem that the expansion and shrinking operation and configuration cannot be performed on the cluster according to the service condition due to lack of experience of the cluster use by a user is avoided, and the efficiency of cluster operation is improved.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario according to the present embodiment. In the application scenario of fig. 3, the management and control service unit 301 may be operated in the execution body, for example, in the server shown in fig. 1, where the management and control service unit 301 may manage the user clusters 302, 303, and 304, and the logic of cluster expansion and contraction may be a module in the management and control service unit 301, and the management and control service unit 301 may select a target cluster from the user clusters 302, 303, and 304, obtain the associated expansion and contraction conditions, and then, in response to detecting that the expansion and contraction conditions are satisfied, perform the expansion and contraction operation corresponding to the expansion and contraction conditions, and finally configure the expansion and contraction operation according to the service type provided by the expansion and contraction operation.
With further reference to fig. 4, a flow 400 of yet another embodiment of a method for cluster scaling is shown. The flow 400 of the method for cluster scaling adds a step of determining the priority of each scaling condition before step 202 of the corresponding embodiment of fig. 2. The process 400 specifically includes the following steps:
step 401, obtaining a capacity expansion and contraction condition associated with a target cluster.
Step 402, determining the priority of each of the capacity expansion conditions.
In this embodiment, a method execution body (e.g., a server shown in fig. 1) for cluster scaling may determine the priority of each of the scaling conditions. The priority may represent the importance degree of the expansion and contraction conditions, and the priority may be determined according to an index such as creation time, or may be determined according to user setting, for example, the user may input the priority of each expansion and contraction condition according to needs.
In some optional implementations of this embodiment, determining the priority of each of the scaling conditions includes: and determining the priority of each expansion and contraction condition in the expansion and contraction conditions according to the creation time, wherein the priority of the expansion and contraction condition with the later creation time is higher. For example, in the four conditions of ABCD, a is created first, B is created after a, C is created after B, D is created last, and DCBA is the priority order from high to low. Because the newly created expansion and contraction conditions may be created by the user aiming at some newly-appearing problems, and are more important than other expansion and contraction conditions, the implementation mode determines the priority of each expansion and contraction condition in the expansion and contraction conditions through the creation time, and preferentially detects the newly-created expansion and contraction conditions during subsequent detection, so that the expansion and contraction flow is further optimized.
Step 403, sequentially detecting each expansion and contraction condition in the expansion and contraction conditions according to the determined priority until the satisfied expansion and contraction conditions are detected.
In this embodiment, the executing body may sequentially detect each of the expansion and contraction conditions according to the determined priority, until the satisfied expansion and contraction condition is detected. For example, the four expansion and contraction conditions of ABCD are sorted from high priority to low priority to be DCBA, whether the rule D is satisfied is preferentially judged, if so, the expansion and contraction operation corresponding to D is executed, and if not, whether the rule B is satisfied is continuously detected.
In step 404, in response to detecting that the capacity expansion and contraction condition is satisfied, a capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition is performed.
In step 405, the type of service provided by the target cluster is determined.
And step 406, configuring the target cluster after the capacity expansion and contraction operation is executed according to the service type.
In this embodiment, the operations of step 401, step 404, step 405, and step 406 are substantially the same as the operations of step 201, step 202, step 203, and step 204, and will not be described herein.
As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, in the flow 400 of the method for expanding and shrinking the cluster in this embodiment, each expansion and shrinking condition in the expansion and shrinking conditions is sequentially detected according to the priority of each expansion and shrinking condition, so that the expansion and shrinking condition with high priority can be responded as soon as possible if satisfied, the expansion and shrinking flow is further optimized, and the efficiency of expanding and shrinking the cluster is improved.
With further reference to fig. 5, a flow 500 of yet another embodiment of a method for cluster scaling is shown. The process 500 of the method for cluster scaling adds a step of starting a user mode thread and triggering a detection task according to the user mode thread before step 202 of the corresponding embodiment of fig. 2. The process 500 specifically includes the following steps:
Step 501, obtaining the expansion and contraction conditions associated with the target cluster.
Step 502, a user mode thread for a target cluster is started.
In this embodiment, a method execution body (e.g., the server shown in fig. 1) for cluster scaling may start a user-mode thread for a target cluster. Since the rules are associated with clusters, one user-mode thread may be started for each cluster.
Step 503, periodically triggering a detection task for detecting the expansion and contraction conditions according to the setting of the user state thread timer.
In this embodiment, the execution body may periodically trigger a detection task for detecting the expansion and contraction condition according to the setting of the user state thread timer. The period of the timer can be set according to actual needs.
In step 504, in response to detecting that the capacity expansion and contraction condition is satisfied, a capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition is performed.
In step 505, the type of service provided by the target cluster is determined.
And step 506, configuring the target cluster after the capacity expansion and contraction operation is executed according to the service type.
In this embodiment, the operations of step 501, step 504, step 505 and step 506 are substantially the same as those of step 201, step 202, step 203 and step 204, and will not be described herein.
As can be seen from fig. 5, compared with the embodiment corresponding to fig. 2, in the process 500 of the method for expanding and shrinking a cluster in this embodiment, a user-mode thread is started for each cluster to manage the related logic of automatic expansion and shrinking, and the efficiency of expanding and shrinking can be further improved based on the triggering of the detection task by the user-mode thread due to the light user-mode thread. With further reference to fig. 6, as an implementation of the method shown in the foregoing drawings, the present application provides an embodiment of an apparatus for cluster expansion and contraction, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.
As shown in fig. 6, the apparatus 600 for cluster expansion and contraction according to the present embodiment includes: an acquisition unit 601, a capacity expansion unit 602, a first determination unit 603, and a configuration unit 604. The acquisition unit is configured to acquire the expansion and contraction conditions associated with the target cluster; a capacity expansion and contraction unit configured to perform capacity expansion and contraction operations corresponding to capacity expansion and contraction conditions in response to detection that the capacity expansion and contraction conditions are satisfied; a first determining unit configured to determine a service type provided by the target cluster; and the configuration unit is configured to configure the target cluster after the expansion and contraction operation is performed according to the service type.
In this embodiment, specific processes of the obtaining unit 601, the scaling unit 602, the first determining unit 603, and the configuring unit 604 of the apparatus 600 for scaling a cluster may refer to steps 201, 202, 203, and 204 in the corresponding embodiment of fig. 2.
In some optional implementations of this embodiment, the apparatus further includes: a second determining unit configured to determine a priority of each of the capacity expansion and contraction conditions; and the detection unit is configured to sequentially detect each expansion and contraction condition in the expansion and contraction conditions according to the determined priority until the satisfied expansion and contraction conditions are detected.
In some optional implementations of the present embodiment, the second determining unit is further configured to: and determining the priority of each expansion and contraction condition in the expansion and contraction conditions according to the creation time, wherein the priority of the expansion and contraction condition with the later creation time is higher.
In some optional implementations of this embodiment, the apparatus further includes: a starting unit configured to start a user mode thread for a target cluster; the trigger unit is configured to periodically trigger a detection task for detecting the expansion and contraction conditions according to the setting of the user state thread timer.
In some optional implementations of the present embodiment, the configuration unit is further configured to: the added resources for the scaling operation in the target cluster are configured with components matching the service type.
The device provided by the embodiment of the application obtains the expansion and contraction conditions associated with the target cluster; in response to detecting that the capacity expansion and contraction conditions are met, performing capacity expansion and contraction operations corresponding to the capacity expansion and contraction conditions; determining the service type provided by the target cluster; and configuring the target cluster after the expansion and contraction operation is executed according to the service type, so that the cluster operation efficiency is improved.
According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
As shown in fig. 7, a block diagram of an electronic device for a method for cluster scaling is provided according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 7, the electronic device includes: one or more processors 701, memory 702, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 701 is illustrated in fig. 7.
Memory 702 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the method for cluster expansion and contraction provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method for cluster expansion and contraction provided by the present application.
The memory 702 is used as a non-transitory computer readable storage medium, and may be used to store a non-transitory software program, a non-transitory computer executable program, and modules, such as program instructions/modules corresponding to the method for scaling a cluster in the embodiment of the present application (e.g., the obtaining unit 501, the scaling unit 502, the first determining unit 503, and the configuring unit 504 shown in fig. 5). The processor 701 executes various functional applications of the server and data processing, i.e., implements the method for cluster expansion and contraction in the above-described method embodiment, by running non-transitory software programs, instructions, and modules stored in the memory 702.
Memory 702 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created from the use of electronic devices for cluster scaling, and the like. In addition, the memory 702 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 702 optionally includes memory remotely located relative to processor 701, which may be connected to the electronic device for cluster expansion and contraction via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device for the method for cluster scaling may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected by a bus or otherwise, in fig. 7 by way of example.
The input device 703 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device for cluster expansion and contraction, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, and the like. The output device 704 may include a display apparatus, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme provided by the embodiment of the application, the cluster operation efficiency is improved.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution disclosed in the present application can be achieved, and are not limited herein.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (11)
1. A method for cluster scaling comprising:
obtaining expansion and contraction conditions associated with a target cluster;
In response to detecting that the capacity expansion and contraction conditions are met, executing capacity expansion and contraction operations corresponding to the capacity expansion and contraction conditions;
determining the service type provided by the target cluster;
Determining components matching the service type for the resources added by the capacity expansion operation in the target cluster; establishing a directed acyclic graph according to the dependency relationship among the components; determining the parallel configuration sequence of each component according to the directed acyclic graph; and configuring each component according to the parallel configuration sequence.
2. The method of claim 1, wherein after the obtaining the target cluster associated scaling conditions, the method further comprises:
determining the priority of each expansion and contraction condition in the expansion and contraction conditions;
and sequentially detecting each expansion and contraction condition in the expansion and contraction conditions according to the determined priority until the satisfied expansion and contraction conditions are detected.
3. The method of claim 2, wherein the determining the priority of each of the scaling conditions comprises:
And determining the priority of each expansion and contraction condition in the expansion and contraction conditions according to the creation time, wherein the later the creation time is, the higher the priority of the expansion and contraction condition is.
4. The method of claim 1, wherein, in response to detecting that the scaling condition is satisfied, prior to performing a scaling operation corresponding to the scaling condition, the method comprises:
starting a user mode thread aiming at the target cluster;
And periodically triggering a detection task for detecting the expansion and contraction conditions according to the setting of the user state thread timer.
5. An apparatus for cluster scaling, comprising:
an acquisition unit configured to acquire a capacity expansion condition associated with a target cluster;
a capacity expansion and contraction unit configured to execute capacity expansion and contraction operation corresponding to the capacity expansion and contraction condition in response to detection that the capacity expansion and contraction condition is satisfied;
a first determining unit configured to determine a type of service provided by the target cluster;
A configuration unit configured to determine components matching the service type for resources added by the scaling operation in the target cluster; establishing a directed acyclic graph according to the dependency relationship among the components; determining the parallel configuration sequence of each component according to the directed acyclic graph; and configuring each component according to the parallel configuration sequence.
6. The apparatus of claim 5, the apparatus further comprising:
A second determining unit configured to determine a priority of each of the expansion and contraction conditions;
And the detection unit is configured to sequentially detect each expansion and contraction condition in the expansion and contraction conditions according to the determined priority until the satisfied expansion and contraction conditions are detected.
7. The apparatus of claim 6, wherein the second determination unit is further configured to:
And determining the priority of each expansion and contraction condition in the expansion and contraction conditions according to the creation time, wherein the later the creation time is, the higher the priority of the expansion and contraction condition is.
8. The apparatus of claim 5, the apparatus further comprising:
A starting unit configured to start a user mode thread for the target cluster;
And the triggering unit is configured to periodically trigger a detection task for detecting the expansion and contraction condition according to the setting of the user state thread timer.
9. An electronic device, comprising:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-4.
11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010614916.1A CN111782147B (en) | 2020-06-30 | 2020-06-30 | Method and device for cluster expansion and contraction capacity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010614916.1A CN111782147B (en) | 2020-06-30 | 2020-06-30 | Method and device for cluster expansion and contraction capacity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111782147A CN111782147A (en) | 2020-10-16 |
| CN111782147B true CN111782147B (en) | 2024-07-26 |
Family
ID=72760521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010614916.1A Active CN111782147B (en) | 2020-06-30 | 2020-06-30 | Method and device for cluster expansion and contraction capacity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111782147B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112506444A (en) * | 2020-12-28 | 2021-03-16 | 南方电网深圳数字电网研究院有限公司 | Kubernetes cluster-based expansion and contraction capacity control method and device and electronic equipment |
| CN112822298B (en) * | 2021-04-19 | 2021-07-23 | 北京世纪好未来教育科技有限公司 | Business service capacity expansion and reduction method, device, medium and electronic equipment |
| CN117806815B (en) * | 2023-11-27 | 2024-07-09 | 本原数据(北京)信息技术有限公司 | Data processing method, system, electronic device and storage medium |
| CN118138591A (en) * | 2024-05-07 | 2024-06-04 | 苏州万店掌网络科技有限公司 | Node management method, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105024842A (en) * | 2014-04-25 | 2015-11-04 | 深圳市腾讯计算机系统有限公司 | Method and device for capacity expansion of server |
| CN109495306A (en) * | 2018-11-27 | 2019-03-19 | 平安科技（深圳）有限公司 | A kind of expansion method and equipment of business network |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317658B (en) * | 2014-10-17 | 2018-06-12 | 华中科技大学 | A kind of loaded self-adaptive method for scheduling task based on MapReduce |
| CN111934904B (en) * | 2014-12-10 | 2023-11-03 | 华为技术有限公司 | Capacity expansion method, controller and system |
| CN104915260B (en) * | 2015-06-19 | 2018-05-25 | 北京搜狐新媒体信息技术有限公司 | A kind of distribution method and system of Hadoop cluster managements task |
| CN105554102A (en) * | 2015-12-14 | 2016-05-04 | 中电科华云信息技术有限公司 | Elastic expansion method based on container cluster and application system thereof |
| CN107302444B (en) * | 2016-04-15 | 2022-03-25 | 中兴通讯股份有限公司 | Enterprise-level search application server cluster automatic capacity expansion method and device |
| CN108076082A (en) * | 2016-11-09 | 2018-05-25 | 阿里巴巴集团控股有限公司 | A kind of expansion method of application cluster, device and system |
| CN106789362B (en) * | 2017-02-20 | 2020-04-14 | 京信通信系统（中国）有限公司 | Equipment management method and network management system |
| CN108769100A (en) * | 2018-04-03 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of implementation method and its device based on kubernetes number of containers elastic telescopics |
-
2020
- 2020-06-30 CN CN202010614916.1A patent/CN111782147B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105024842A (en) * | 2014-04-25 | 2015-11-04 | 深圳市腾讯计算机系统有限公司 | Method and device for capacity expansion of server |
| CN109495306A (en) * | 2018-11-27 | 2019-03-19 | 平安科技（深圳）有限公司 | A kind of expansion method and equipment of business network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111782147A (en) | 2020-10-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111782147B (en) | Method and device for cluster expansion and contraction capacity | |
| TWI773100B (en) | Neural network structure search method and apparatus | |
| US9942353B2 (en) | Management of connections within a messaging environment based on the statistical analysis of server responsiveness | |
| JP7214786B2 (en) | Scheduling method, device, device and medium for deep learning inference engine | |
| CN111694646B (en) | Resource scheduling method, device, electronic equipment and computer readable storage medium | |
| CN111782341B (en) | Method and device for managing clusters | |
| CN110806923A (en) | Parallel processing method and device for block chain tasks, electronic equipment and medium | |
| CN111488492B (en) | Method and device for searching graph database | |
| CN111506401B (en) | Automatic driving simulation task scheduling method and device, electronic equipment and storage medium | |
| CN111880914A (en) | Resource scheduling method, resource scheduling apparatus, electronic device, and storage medium | |
| CN112527474A (en) | Task processing method and device, equipment, readable medium and computer program product | |
| CN111782357B (en) | Label control method and device, electronic equipment and readable storage medium | |
| CN112817992A (en) | Method, device, electronic equipment and readable storage medium for executing change task | |
| US10594620B1 (en) | Bit vector analysis for resource placement in a distributed system | |
| CN113760638A (en) | Log service method and device based on kubernets cluster | |
| CN113778973B (en) | Data storage method and device | |
| CN111597026B (en) | Method and device for acquiring information | |
| CN111416860B (en) | Transaction processing method and device based on block chain, electronic equipment and medium | |
| CN110515622B (en) | Button state control method and device, electronic equipment and storage medium | |
| CN112148563B (en) | Information generation method, device, equipment and storage medium | |
| CN111258954B (en) | Data migration method, device, equipment and storage medium | |
| CN112437152B (en) | Crash processing method and device, electronic equipment and storage medium | |
| US20170134238A1 (en) | Physical machine management device and physical machine management method | |
| CN111488366B (en) | Relational database updating method, relational database updating device, relational database updating equipment and storage medium | |
| US11099780B2 (en) | Adaptively tuning polling intervals of control hosts in distributed storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |