CN114138451A - Cluster deployment method and device, disk allocation method, electronic device and medium - Google Patents
Cluster deployment method and device, disk allocation method, electronic device and medium Download PDFInfo
- Publication number
- CN114138451A CN114138451A CN202010917313.9A CN202010917313A CN114138451A CN 114138451 A CN114138451 A CN 114138451A CN 202010917313 A CN202010917313 A CN 202010917313A CN 114138451 A CN114138451 A CN 114138451A
- Authority
- CN
- China
- Prior art keywords
- physical host
- cloud physical
- instances
- disks
- cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Abstract
The disclosure relates to a cluster deployment method, a cluster deployment device, a disk allocation method, an electronic device and a medium. The method comprises the following steps: determining configuration parameters, wherein the configuration parameters comprise the number of instances of a cluster and the role of the instances; determining the number of examples matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters; distributing a CPU core for the example on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched with each cloud physical host; and performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core. The embodiment of the disclosure improves the utilization rate of resources and balances the read-write performance.
Description
Technical Field
The present disclosure relates to the field of cluster deployment technologies, and in particular, to a cluster deployment method and apparatus, a disk allocation method, an electronic device, and a medium.
Background
The Elasticissearch is a distributed search and analysis engine which is constructed based on Lucene and supports RESTful API. The method has the advantages of low memory consumption, fast search, strong expansibility and the like, and can provide the capability of storing, searching and analyzing the super-large data set in near real time.
For hosting an Elasticsearch, for a small data scale scene with low performance requirement, a virtual machine can be used for establishing an Elasticsearch cluster. But for low latency, high concurrency, and high throughput scenarios, deploying an Elasticsearch cluster using a highly configured cloud Physical host (EPC), is advantageous in both performance and cost.
Currently, only one ES (elastic search) instance is started on each cloud physical host in the elastic search cluster, which exclusively occupies resources of the cloud physical host except for the memory, the heap memory of the ES instance is smaller than 32G, and the rest of the memory is reserved for the Lucence cache. This mode, while providing greater probability of a hit to the cache for a read, increases query speed. However, for a scenario (such as log query) where there are more writes and fewer reads or the timeliness of the read operation is not so high, such a mode may cause unreasonable resource allocation and reduce the cluster resource utilization.
Disclosure of Invention
To solve the technical problem or at least partially solve the technical problem, the present disclosure provides a cluster deployment method, an apparatus, a disk allocation method, an electronic device, and a medium.
The present disclosure provides a cluster deployment method, including:
determining configuration parameters, wherein the configuration parameters comprise the number of instances of a cluster and the role of the instances;
determining the number of examples matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters;
distributing a CPU core for the example on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched with each cloud physical host;
and performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core.
Optionally, determining the number of instances matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters includes:
determining the maximum number of instances which can be started by each cloud physical host based on the memory size of each cloud physical host and a preset memory allocation mode;
and determining the number of the instances matched with each cloud physical host as a target number based on the maximum number of the instances which can be started by each cloud physical host and the number of the instances of the cluster, wherein the target number is less than or equal to the maximum number of the instances which can be started by the corresponding cloud physical host.
Optionally, the instance role includes a master node instance, a data instance, and a coordinator instance, where the master node instance is used to manage metadata, the data instance is used to store data, and the coordinator instance is used to distribute data;
the number of the host node instances deployed in the cluster is odd and is greater than or equal to 3, and the number of the host node instances matched with each cloud physical host is less than or equal to 1.
Optionally, allocating the CPU core for the instance on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the instances corresponding to the matching on each cloud physical host includes:
if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is an integer, evenly distributing the CPU cores for the examples on the corresponding cloud physical host;
and if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is not an integer, taking the integer part of the ratio, distributing the number of the CPU cores corresponding to the integer part to each example on the corresponding cloud physical host, and distributing the rest CPU cores to part of the examples.
Optionally, before performing cluster deployment, the method further includes:
creating a separate data directory for each instance in all disks of the cloud physical host;
performing cluster deployment, including:
and according to the number of the matched examples of each cloud physical host, the CPU core and the data directory created for each example, carrying out cluster deployment.
Optionally, before performing cluster deployment, the method further includes:
if the number of the disks of the cloud physical host is greater than or equal to the number of the ES instances on the cloud physical host, distributing a part of disks on the corresponding cloud physical host for each instance on the cloud physical host, wherein the disks distributed by different instances are not overlapped;
performing cluster deployment, including:
and carrying out cluster deployment according to the number of the matched examples of each cloud physical host, the CPU core and the disk distributed to each example.
Optionally, allocating a part of disks on the corresponding cloud physical host to each instance on the cloud physical host includes:
if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is an integer, evenly distributing the disks for the instances on the corresponding cloud physical host;
and if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is not an integer, taking the integer part of the ratio, distributing the number of the disks corresponding to the integer part to each instance on the corresponding cloud physical host, and distributing the rest of the disks to the partial instances.
Optionally, the matching instance on the cloud physical host is obtained through virtual machine simulation on the cloud physical host.
The present disclosure provides a disk allocation method, including:
determining the number of disks of the cloud physical host and the number of matched examples;
and distributing the disks for each instance based on the size relationship between the number of the disks and the number of the matched instances.
The present disclosure provides a cluster deployment device, comprising:
the parameter determining module is used for determining configuration parameters, wherein the configuration parameters comprise the number of instances of the cluster and the role of the instances;
the example determining module is used for determining the number of examples matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters;
the core distribution module is used for distributing the CPU core for the example on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched with each cloud physical host;
and the cluster deployment module is used for performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core.
The present disclosure provides an electronic device, the electronic device including:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the cluster deployment method and/or the disk allocation method provided by the present disclosure.
The present disclosure provides a computer-readable storage medium storing a computer program for executing the cluster deployment method and/or the disk allocation method provided by the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
the technical scheme provided by the embodiment of the disclosure includes that a plurality of cloud physical hosts are adopted to deploy a cluster, the number of examples matched with each cloud physical host is determined based on the memory size and configuration parameters of each cloud physical host, and meanwhile, the number of CPU cores is distributed to the examples on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched on each corresponding cloud physical host. On one hand, the CPU cores are distributed to the examples on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the matched examples corresponding to each cloud physical host, so that the utilization rate of the memory resources is improved. On the other hand, each cloud physical host can be matched with a plurality of instances, so that the off-heap memory of the cloud physical host is relatively reduced, and the read-write performance is balanced; meanwhile, the memory of each instance heap can be set to be less than or equal to 32G, the memory object pointer compression technology can be utilized, the waste of the memory is avoided, the large memory corresponding to Java garbage recovery is avoided, the consumption of a CPU is reduced, and the performance of the cluster is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flow chart of a cluster deployment method provided in the embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of a cluster provided in an embodiment of the present disclosure;
fig. 3 is a block diagram of a cluster deployment apparatus provided in the embodiment of the present disclosure;
fig. 4 is a schematic flowchart of a disk allocation method according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Fig. 1 is a schematic flow chart of a cluster deployment method provided in the embodiment of the present disclosure. The cluster deployment method is applicable to the elastic search service and other search and analysis engine services, and can be executed by a cluster deployment device, wherein the cluster deployment device can be realized in a software and/or hardware manner, and can be generally integrated in electronic equipment. In the embodiment of the present disclosure, the cluster may be an Elasticsearch cluster, the Elasticsearch cluster is configured with a plurality of cloud physical hosts, and accordingly, in the deployment process of the Elasticsearch cluster, an example matched to each cloud physical host is an Elasticsearch example (abbreviated as an ES example). As shown in fig. 1, the cluster deployment method includes:
In some embodiments of the present disclosure, a client initiates an HTTP request to a server, where the HTTP request includes the number of instances and the role of the instances of a cluster to be created; and the server receives the HTTP request, analyzes the HTTP request, and then sends the analyzed HTTP request to the cloud physical host, so as to determine the configuration parameters.
And step 120, determining the number of the examples matched with each cloud physical host based on the memory size and the configuration parameters of each cloud physical host.
In some embodiments of the present disclosure, an agent process is started on a cloud physical host, and the agent process may determine the number of instances based on the memory size of the cloud physical host where the agent process is located and the acquired configuration parameters, where an instance refers to a process.
First, the number of matching instances on each cloud physical host may be determined. Optionally, the maximum number of instances that each cloud physical host can start is determined based on the memory size of each cloud physical host and a preset memory allocation mode; and determining the number of the matched examples on each cloud physical host as a target number based on the maximum number of the examples which can be started by each cloud physical host and the number of the examples of the cluster, wherein the target number is less than or equal to the maximum number of the examples which can be started by the corresponding cloud physical host. Optionally, the number of targets is multiple, that is, multiple instances are matched on each cloud physical host, so that the off-heap memory of the cloud physical host is relatively reduced, and the read-write performance is balanced.
Meanwhile, when the memory of the java virtual machine is smaller than 32G, a memory object pointer compression technology is adopted to solve the problems of memory waste and more bandwidth occupation. When the 32G boundary is crossed, the pointer will switch back to the pointer of the normal object, so that the pointer of each object becomes longer, and thus more CPU memory bandwidth is used, i.e. more memory is actually lost. As a matter of fact, when the memory reaches 40-50GB, the effective memory is equivalent to 32G memory when the memory object pointer compression technique is used. Therefore, according to the scheme, the instance heap memory is set to be less than or equal to 32G, the memory object pointer compression technology can be used, the waste of the memory is avoided, the large memory corresponding to Java garbage recovery is avoided, the consumption of a CPU is reduced, and the performance of a cluster is improved.
For example, the preset memory allocation manner may be: the sum of the heap memory and the out-of-heap memory corresponding to each instance is 64G, and the heap memory is less than or equal to 32G. Therefore, the maximum number of instances which can be started by the cloud physical host is obtained by the ratio of the memory of the cloud physical host to 64. For example, if the memory of the cloud physical host is 256G, the maximum number of instances that can be started by the cloud physical host is 256/64-4. When the number of the instances needing to be started by the cloud physical host is determined based on the number of the instances of the cluster and is equal to the maximum number of the instances which can be started by the cloud physical host, the heap memory and the external memory are directly allocated according to a preset memory allocation mode. When the number of the instances needing to be started of the cloud physical host is determined based on the number of the instances of the cluster and is smaller than the maximum number of the instances which can be started by the cloud physical host, the heap memory can be distributed according to a preset memory distribution mode, the memory mean value is obtained according to the ratio of the memory of the cloud physical host to the number of the instances needing to be started, and finally the out-of-heap memory is obtained according to the difference value of the memory mean value and the heap memory. For example, the cloud physical host needs to start 2 instances, the heap memory size of each instance is 31G, the average memory value is 256/2-128G, and the out-of-heap memory corresponding to each instance is 128-31-97G.
The role of the launched instance on each cloud physical host can then be determined. Optionally, the instance role may include a master node instance, a data instance, and a coordinator instance, where the master node instance is used to manage metadata, the data instance is used to store data, and the coordinator instance is used to distribute data; the number of the main node instances deployed in the cluster is odd and is greater than or equal to 3, and the number of the main node instances matched with each cloud physical host is less than or equal to 1.
The number of the host node instances is odd and is more than or equal to 3, and the number of the host node instances matched with each cloud physical host is less than or equal to 1, so that at least 3 cloud physical hosts are configured in the cluster; meanwhile, the number of the main node examples is set to be odd and greater than or equal to 3, and the number of the main node examples matched with each cloud physical host is less than or equal to 1, so that split brains can be avoided, the phenomenon that the whole cluster is unavailable due to the fault of one cloud physical host is prevented, and a high-availability cluster is constructed; additionally, the instance role of the cluster necessarily includes the master node instance and the data instance. Illustratively, referring to fig. 2, the cluster 20 is configured with 5 cloud physical hosts, including a first cloud physical host 201, a second cloud physical host 202, a third cloud physical host 203, a fourth cloud physical host 204, and a fifth cloud physical host 205. Each cloud physical host needs to match 4 instances, and the role of the cluster instance only includes a master node instance and data instances, where the master node instance is 3, 1 master node instance and 3 data instances may be respectively matched on the first cloud physical host 201, the second cloud physical host 202, and the third cloud physical host 203, and 4 data instances may be respectively matched on the fourth cloud physical host 204 and the fifth cloud physical host 205.
To ensure full utilization of resources, all CPU cores of the cloud physical host are allocated to the instance. In some embodiments of the present disclosure, if the ratio of the number of CPU cores of a cloud physical host to the number of instances matched by a corresponding cloud physical host is an integer, the CPU cores are evenly allocated to the instances on the corresponding cloud physical host. For example, a cloud physical host includes 16 CPU cores, and if there are 4 matching instances on the cloud physical host, there are 4 CPU cores allocated to each instance on the cloud physical host.
In some embodiments of the present disclosure, if the ratio of the number of CPU cores of the cloud physical host to the number of matched instances on the corresponding cloud physical host is not an integer, an integer part of the ratio is taken, the number of CPU cores corresponding to the integer part is allocated to each instance on the corresponding cloud physical host, and the remaining CPU cores are allocated to part of the instances. Wherein, allocating the remaining CPU cores to part of the instances may include: allocating the remaining CPU cores to different instances; or allocating the remaining CPU cores to the same instance; or a part of the remaining CPU cores may be allocated to the same instance and another part to other instances. Specifically, when the remaining CPU cores are allocated, the remaining CPU cores may be allocated to the instances according to the sequence of the instances when the starting sequence is preset. For example, the cloud physical host includes 16 CPU cores, 3 instances are started on the cloud physical host, the number of the CPU cores allocated to the 3 instances is 5, and 6, respectively, and for convenience of allocation, the remaining CPU cores (the remaining 1 in this example) may be allocated to instances that are preset to be started first or started last.
And 140, performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core.
The step is based on the number of the matched examples of each cloud physical host determined in the above steps 110 to 130 and the CPU core allocated to the ES example on each cloud physical host, and completes the cluster deployment.
The technical scheme provided by the embodiment of the disclosure includes that a plurality of cloud physical hosts are adopted to deploy a cluster, a plurality of examples are started on each corresponding cloud physical host based on the memory size and configuration parameters of each cloud physical host, the memory of each example is less than or equal to 32G, and the CPU core is distributed to the examples on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples started on each corresponding cloud physical host. On one hand, the memory of each instance heap is set to be less than or equal to 32G, the memory object pointer compression technology can be utilized, the waste of the memory is avoided, the large memory corresponding to Java garbage recovery is avoided, the consumption of a CPU is reduced, and the performance of a cluster is improved. On the other hand, each cloud physical host starts a plurality of instances, so that the off-heap memory of the cloud physical host is relatively reduced, and the read-write performance is balanced; and meanwhile, the CPU cores are distributed to the examples on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples started on each corresponding cloud physical host, so that the utilization rate of the memory resources is improved.
Based on the technical scheme, the embodiment of the disclosure also provides an example disk allocation mode to further improve the utilization rate of resources.
In some embodiments of the present disclosure, before performing cluster deployment, the method further includes:
creating a separate data directory for each instance in all disks of the cloud physical host;
correspondingly, cluster deployment is carried out, and the cluster deployment comprises the following steps: and according to the number of the matched examples of each cloud physical host, the CPU core and the data directory created for each example, carrying out cluster deployment.
Therefore, by creating the data directories of the instances in all the disks, the number of the disks of the cloud physical host is not required to be considered, and the instances can be distributed to the disks under the conditions of reducing the number of the disks and saving disk resources.
In addition, in consideration of the fact that the disks are allocated for the instances in a mode that a separate data directory is created for each instance in all the disks of the cloud physical host, when any one disk fails, all the instances can be affected. Therefore, in some embodiments of the present disclosure, before performing cluster deployment, the method may further include:
if the number of the disks of the cloud physical host is greater than or equal to the number of the ES instances on the cloud physical host, distributing a part of disks on the corresponding cloud physical host for each instance on the cloud physical host, wherein the disks distributed by different instances are not overlapped;
correspondingly, cluster deployment is carried out, and the cluster deployment comprises the following steps: and carrying out cluster deployment according to the number of the matched examples of each cloud physical host, the CPU core and the disk distributed to each example.
Compared with the former disk allocation mode, the scheme is more suitable for the condition that the number of disks of the cloud physical host is larger than the number of instances on the cloud physical host, and by designing each instance to share part of the disks independently, even if the disks are damaged, the usability of the cluster is not influenced by matching with a copy mechanism of the instance.
In the foregoing technical solution, allocating a part of disks on a corresponding cloud physical host to each instance on the cloud physical host may include:
if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is an integer, evenly distributing the disks for the instances on the corresponding cloud physical host;
if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is not an integer, taking the integer part of the ratio, allocating the number of the disks corresponding to the integer part to each instance on the corresponding cloud physical host, and allocating the rest of the disks to the partial instances.
In this scheme, the allocation method for allocating disks to instances is similar to the allocation method for allocating CPU cores to instances in the above scheme, and reference may be made to the above scheme. Therefore, the number of disks owned by each instance is equal or nearly equal (the maximum difference value is smaller than the number of the instances matched with the cloud physical host), the IO resources of the disks among the instances are kept consistent, and other problems caused by uneven resources are avoided.
Additionally, in any of the embodiments provided in the present disclosure, the matching instance on the cloud physical host may be obtained through a virtual machine simulation on the cloud physical host.
It can be understood that the specific implementation manner of subsequently starting multiple instances is not limited in the embodiments of the present disclosure, and the instances may be simulated using virtualization technology, or may be directly created on a cloud physical host.
Fig. 3 is a block diagram of a cluster deployment device according to an embodiment of the present disclosure. Wherein, the cluster is configured with a plurality of cloud physical hosts, as shown in fig. 3, the cluster deployment apparatus includes:
a parameter determining module 301, configured to determine configuration parameters, where the configuration parameters include the number of instances of the cluster and the role of the instances;
an example determining module 302, configured to determine, based on the memory size and the configuration parameters of each cloud physical host, the number of examples that are matched with each cloud physical host;
a core allocation module 303, configured to allocate a CPU core to an instance on each cloud physical host based on the number of CPU cores of each cloud physical host and the number of instances corresponding to each cloud physical host;
and the cluster deployment module 304 is configured to perform cluster deployment according to the number of the instances and the CPU cores matched with each cloud physical host.
Optionally, the example starting module 302 includes:
the maximum number determining unit is used for determining the maximum number of the starting examples of each cloud physical host based on the memory size of each cloud physical host and a preset memory allocation mode;
and the example starting unit is used for determining the number of the matched examples on each cloud physical host as a target number based on the maximum number of the examples which can be started by each cloud physical host and the number of the examples of the cluster, wherein the target number is less than or equal to the maximum number of the examples which can be started by the corresponding cloud physical host.
Optionally, the instance role includes a master node instance, a data instance, and a coordinator instance, where the master node instance is used to manage metadata, the data instance is used to store data, and the coordinator instance is used to distribute data;
the number of the main node instances deployed in the cluster is odd and is greater than or equal to 3, and the number of the main node instances matched with each cloud physical host is less than or equal to 1.
Optionally, the core allocating module 303 includes:
the first core distribution unit is used for evenly distributing the CPU cores for the examples on the corresponding cloud physical host if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is an integer;
and the second core allocation unit is used for taking the integer part of the ratio if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is not an integer, allocating the number of the CPU cores corresponding to the integer part to each example on the corresponding cloud physical host, and allocating the rest CPU cores to part of the examples.
Optionally, the cluster deploying apparatus further includes:
the first disk allocation module is used for creating an independent data directory for each instance in all disks of the cloud physical host;
correspondingly, the cluster deployment module 304 is specifically configured to perform cluster deployment according to the number of instances matched by each cloud physical host, the CPU core, and a data directory created for each instance.
Optionally, the cluster deploying apparatus further includes:
the second disk allocation module is used for allocating a part of disks on the corresponding cloud physical host to each instance on the cloud physical host if the number of the disks of the cloud physical host is greater than or equal to the number of the ES instances on the cloud physical host, wherein the disks allocated to different instances are not overlapped;
correspondingly, the cluster deployment module 304 is specifically configured to perform cluster deployment according to the number of instances matched by each cloud physical host, the CPU core, and the disk allocated to each instance.
Optionally, the second disk allocation module includes:
the first disk allocation unit is used for evenly allocating disks to the examples on the corresponding cloud physical host if the ratio of the number of the disks on the cloud physical host to the number of the examples on the corresponding cloud physical host is an integer;
and the second disk allocation unit is used for taking the integer part of the ratio if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is not an integer, allocating the disks with the number corresponding to the integer part to each instance on the corresponding cloud physical host, and allocating the rest of the disks to the partial instances.
Alternatively, the above example can be obtained by virtual machine simulation on the cloud physical host.
The cluster deployment device provided by the embodiment of the disclosure can execute the cluster deployment method provided by any embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method. For the content that is not described in detail in the embodiments of the apparatus of the present disclosure, please refer to the embodiments of the method of the present disclosure, which will not be described herein again.
The embodiment of the disclosure further provides a disk allocation method, and fig. 4 is a schematic flow diagram of the disk allocation method provided by the embodiment of the disclosure. As shown in fig. 4, the disk allocation method includes:
and step 210, determining the number of disks of the cloud physical host and the number of matched examples.
After the cloud physical host is determined, the number of the disks of the cloud physical host is a known fixed value. In some embodiments, the number of instances that match each cloud physical host may be determined based on the memory size of each cloud physical host and configuration parameters.
Specifically, an HTTP request is sent to a server based on a client, and configuration parameters including the number of instances of a cluster and the roles of the instances are analyzed; determining the maximum number of instances which can be started by each cloud physical host based on the memory size of each cloud physical host and a preset memory allocation mode; and determining the number of the matched examples on each cloud physical host as a target number based on the maximum number of the examples which can be started by each cloud physical host and the number of the examples of the cluster, wherein the target number is less than or equal to the maximum number of the examples which can be started by the corresponding cloud physical host.
And step 220, distributing the disks for each example based on the size relationship between the number of the disks and the number of the matched examples.
According to the configuration of different cloud physical hosts and the number of the matched instances on the cloud physical host under the actual request of the client, the number of the disks is possibly greater than, equal to or less than the number of the matched instances on the cloud physical host.
In some embodiments, a separate data directory may be created for each instance in all disks of the cloud physical host, whether the number of disks is greater than, equal to, or less than the number of matching instances on the cloud physical host. Therefore, by creating the data directories of the instances in all the disks, the number of the disks of the cloud physical host is not required to be considered, the number of the disks can be reduced, and the disks can be allocated to the instances under the condition of saving disk resources, so that the utilization rate of the resources is improved.
In addition, in consideration of the fact that the disks are allocated for the instances in a mode that a separate data directory is created for each instance in all the disks of the cloud physical host, when any one disk fails, all the instances can be affected. Therefore, in some preferred embodiments of the present disclosure, if the number of disks of the cloud physical host is greater than or equal to the number of ES instances on the cloud physical host, each instance on the cloud physical host is allocated with a portion of disks on the corresponding cloud physical host, where the disks allocated to different instances do not overlap. Compared with the former disk allocation mode, the scheme is more suitable for the condition that the number of disks of the cloud physical host is larger than the number of instances on the cloud physical host, and by designing each instance to share part of the disks independently, even if the disks are damaged, the usability of the cluster is not influenced by matching with a copy mechanism of the instance.
An embodiment of the present disclosure provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and the processor is used for reading the executable instructions from the memory and executing the instructions to realize the cluster deployment method provided by the embodiment of the disclosure.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 5, the electronic device 400 includes one or more processors 401 and memory 402.
The processor 401 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.
In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 403 may also include, for example, a keyboard, a mouse, and the like.
The output device 404 may output various information to the outside, including the determined distance information, direction information, and the like. The output devices 404 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 400 relevant to the present disclosure are shown in fig. 5, omitting components such as buses, input/output interfaces, and the like. In addition, electronic device 400 may include any other suitable components depending on the particular application.
In addition to the above methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the cluster deployment methods provided by embodiments of the present disclosure.
The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the cluster deployment method provided by embodiments of the present disclosure.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. A method for cluster deployment, comprising:
determining configuration parameters, wherein the configuration parameters comprise the number of instances of a cluster and the role of the instances;
determining the number of examples matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters;
distributing a CPU core for the example on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched with each cloud physical host;
and performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core.
2. The cluster deployment method of claim 1, wherein determining the number of instances that match each cloud physical host based on the memory size of each cloud physical host and the configuration parameters comprises:
determining the maximum number of instances which can be started by each cloud physical host based on the memory size of each cloud physical host and a preset memory allocation mode;
and determining the number of the instances matched with each cloud physical host as a target number based on the maximum number of the instances which can be started by each cloud physical host and the number of the instances of the cluster, wherein the target number is less than or equal to the maximum number of the instances which can be started by the corresponding cloud physical host.
3. The cluster deployment method of claim 2, wherein the instance roles comprise a master node instance for managing metadata, a data instance for storing data, and a coordinator instance for distributing data;
the number of the host node instances deployed in the cluster is odd and is greater than or equal to 3, and the number of the host node instances matched with each cloud physical host is less than or equal to 1.
4. The cluster deployment method of claim 1, wherein allocating CPU cores for the instances on each of the cloud physical hosts based on the number of CPU cores per cloud physical host and the number of instances matching for each cloud physical host comprises:
if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is an integer, evenly distributing the CPU cores for the examples on the corresponding cloud physical host;
and if the ratio of the number of the CPU cores of the cloud physical host to the number of the matched examples on the corresponding cloud physical host is not an integer, taking the integer part of the ratio, distributing the number of the CPU cores corresponding to the integer part to each example on the corresponding cloud physical host, and distributing the rest CPU cores to part of the examples.
5. The cluster deployment method of claim 1, further comprising, before the cluster deployment, the steps of:
creating a separate data directory for each instance in all disks of the cloud physical host;
performing cluster deployment, including:
and according to the number of the matched examples of each cloud physical host, the CPU core and the data directory created for each example, carrying out cluster deployment.
6. The cluster deployment method of claim 1, further comprising, before the cluster deployment, the steps of:
if the number of the disks of the cloud physical host is greater than or equal to the number of the ES instances on the cloud physical host, distributing a part of disks on the corresponding cloud physical host for each instance on the cloud physical host, wherein the disks distributed by different instances are not overlapped;
performing cluster deployment, including:
and carrying out cluster deployment according to the number of the matched examples of each cloud physical host, the CPU core and the disk distributed to each example.
7. The cluster deployment method of claim 6, wherein allocating a portion of the disks on the corresponding cloud physical host for each instance on the cloud physical host comprises:
if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is an integer, evenly distributing the disks for the instances on the corresponding cloud physical host;
and if the ratio of the number of the disks on the cloud physical host to the number of the instances on the corresponding cloud physical host is not an integer, taking the integer part of the ratio, distributing the number of the disks corresponding to the integer part to each instance on the corresponding cloud physical host, and distributing the rest of the disks to the partial instances.
8. The cluster deployment method of claim 1, wherein the matching instances on the cloud physical host are obtained through virtual machine simulation on the cloud physical host.
9. A method for allocating disks, comprising:
determining the number of disks of the cloud physical host and the number of matched examples;
and distributing the disks for each instance based on the size relationship between the number of the disks and the number of the matched instances.
10. A cluster deployment apparatus, comprising:
the parameter determining module is used for determining configuration parameters, wherein the configuration parameters comprise the number of instances of the cluster and the role of the instances;
the example determining module is used for determining the number of examples matched with each cloud physical host based on the memory size of each cloud physical host and the configuration parameters;
the core distribution module is used for distributing the CPU core for the example on each cloud physical host based on the number of the CPU cores of each cloud physical host and the number of the examples matched with each cloud physical host;
and the cluster deployment module is used for performing cluster deployment according to the number of the matched examples of each cloud physical host and the CPU core.
11. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the cluster deployment method according to any one of claims 1 to 8 and/or the disk allocation method according to claim 9.
12. A computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the cluster deployment method of any of the above claims 1-8 and/or the disk allocation method of claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010917313.9A CN114138451A (en) | 2020-09-03 | 2020-09-03 | Cluster deployment method and device, disk allocation method, electronic device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010917313.9A CN114138451A (en) | 2020-09-03 | 2020-09-03 | Cluster deployment method and device, disk allocation method, electronic device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114138451A true CN114138451A (en) | 2022-03-04 |
Family
ID=80438393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010917313.9A Pending CN114138451A (en) | 2020-09-03 | 2020-09-03 | Cluster deployment method and device, disk allocation method, electronic device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114138451A (en) |
-
2020
- 2020-09-03 CN CN202010917313.9A patent/CN114138451A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10871960B2 (en) | Upgrading a storage controller operating system without rebooting a storage system | |
| JP5680961B2 (en) | Integrated provision of physical and virtual images | |
| US20110078681A1 (en) | Method and system for running virtual machine image | |
| US10740148B2 (en) | Accelerated data operations in virtual environments | |
| US11809901B2 (en) | Migrating the runtime state of a container between two nodes | |
| KR20090079012A (en) | Method and apparatus for save/restore state of virtual machine | |
| US20110202918A1 (en) | Virtualization apparatus for providing a transactional input/output interface | |
| US8732427B2 (en) | Systems and methods for collapsing a derivative version of a primary storage volume | |
| US10817205B2 (en) | Computer system and storage device | |
| KR102326280B1 (en) | Method, apparatus, device and medium for processing data | |
| US20140082275A1 (en) | Server, host and method for reading base image through storage area network | |
| US10579419B2 (en) | Data analysis in storage system | |
| JP2021515299A (en) | A system for chronological out-of-place updates, a method for chronological out-of-place updates, and a computer program for chronological out-of-place updates. | |
| US20130290276A1 (en) | Enhancing performance-cost ratio of a primary storage adapative data reduction system | |
| CN107329798B (en) | Data replication method and device and virtualization system | |
| US9158550B2 (en) | Caching based operating system installation | |
| CN116028455A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN115562871A (en) | Memory allocation management method and device | |
| US9256648B2 (en) | Data handling in a cloud computing environment | |
| CN114138451A (en) | Cluster deployment method and device, disk allocation method, electronic device and medium | |
| Caldwell et al. | Fluidmem: Memory as a service for the datacenter | |
| US20220318042A1 (en) | Distributed memory block device storage | |
| KR101754713B1 (en) | Asymmetric distributed file system, apparatus and method for distribution of computation | |
| CN113448897B (en) | Optimization method suitable for pure user mode far-end direct memory access | |
| AU2021265928B2 (en) | Transactional memory based memory page de-duplication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |