WO2023185355A1 - 实现集群虚拟机高可用的方法和装置、设备和介质 - Google Patents

实现集群虚拟机高可用的方法和装置、设备和介质 Download PDF

Info

Publication number
WO2023185355A1
WO2023185355A1 PCT/CN2023/078674 CN2023078674W WO2023185355A1 WO 2023185355 A1 WO2023185355 A1 WO 2023185355A1 CN 2023078674 W CN2023078674 W CN 2023078674W WO 2023185355 A1 WO2023185355 A1 WO 2023185355A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual machine
availability
recovery
status
strategy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/078674
Other languages
English (en)
French (fr)
Inventor
陈志强
张丽
张艺钟
唐虎
李曦
宋润坤
于大钊
汪汀
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nuctech Co Ltd
Original Assignee
Nuctech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202210511479.XA external-priority patent/CN114691304B/zh
Application filed by Nuctech Co Ltd filed Critical Nuctech Co Ltd
Publication of WO2023185355A1 publication Critical patent/WO2023185355A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operations
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1415—Saving, restoring, recovering or retrying at system level
    • G06F11/1438—Restarting or rejuvenating
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0712—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a virtual computing platform, e.g. logically partitioned systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0751—Error or fault detection not based on redundancy
    • G06F11/0754—Error or fault detection not based on redundancy by exceeding limits
    • G06F11/0757—Error or fault detection not based on redundancy by exceeding limits by exceeding a time limit, i.e. time-out, e.g. watchdogs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operations
    • G06F11/1479—Generic software techniques for error detection or fault masking
    • G06F11/1482—Generic software techniques for error detection or fault masking using middleware or operating system [OS] functionalities
    • G06F11/1484—Generic software techniques for error detection or fault masking using middleware or operating system [OS] functionalities involving virtual machines
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/815—Virtual

Definitions

  • the present disclosure relates to the field of cloud computing technology, and specifically to a method and device for realizing high availability of cluster virtual machines, electronic equipment, computer-readable storage media and program products.
  • OpenStack As the cloud computing industry continues to rise, OpenStack is widely used in all walks of life and has gradually become the mainstream virtual machine management software for major companies. But OpenStack itself does not have a complete set of virtual machine high availability solutions. When a virtual machine fails, the virtual machine often cannot recover on its own without human intervention, resulting in the unavailability of applications running on the virtual machine and affecting the entire cloud business.
  • a method for achieving high availability of clustered virtual machines including:
  • the high-availability recovery strategy includes at least one of a first high-availability recovery strategy and a second high-availability recovery strategy, the first high-availability recovery strategy is a restart/startup strategy, and the The second highest availability recovery strategy is the rebuild strategy.
  • performing the recovery operation on the at least one virtual machine includes:
  • performing the recovery operation on the at least one virtual machine includes:
  • the at least one virtual machine is rebuilt on the host machine where the at least one virtual machine is located.
  • the high-availability recovery strategy further includes a third high-availability recovery strategy.
  • the at least one virtual machine-specific high-availability recovery strategy is the third high-availability recovery strategy
  • the at least one virtual machine-specific high-availability recovery strategy is Recovery operations performed by a virtual machine include:
  • the at least one virtual machine is rebuilt on the host machine where the at least one virtual machine is located.
  • performing the recovery operation on the at least one virtual machine further includes:
  • performing the recovery operation on the at least one virtual machine further includes:
  • the at least one virtual machine is rebuilt on another host machine, wherein the other host machines and the host machine where the at least one virtual machine is located are on the same available network. domain.
  • rebuilding the at least one virtual machine on other hosts includes:
  • performing the recovery operation on the at least one virtual machine further includes:
  • the high-availability task queue is updated.
  • the high availability task queue includes an error task queue
  • the error task queue includes a virtual machine queue that has failed in high availability recovery
  • the high-availability task queue is updated based on the results of the above judgment, specifically including:
  • the at least one virtual machine is added to the error task queue.
  • the high-availability task queue further includes a running task queue
  • the running task queue includes a virtual machine queue that is currently running or waiting to run a high-availability recovery operation
  • the high-availability task queue is updated based on the results of the above judgment, specifically including:
  • the at least one virtual machine When the state of the at least one virtual machine returns to normal, the at least one virtual machine is removed from the running task queue.
  • the high-availability recovery strategy further includes a fourth high-availability recovery strategy, and the fourth high-availability recovery strategy is an ignore strategy,
  • the method also includes:
  • the high-availability recovery policy specific to the at least one virtual machine is the fourth high-availability recovery policy, no recovery operation is performed on the at least one virtual machine.
  • the high-availability task queue further includes a ignore task queue
  • the ignore task queue includes a virtual machine queue with a dedicated high-availability recovery policy configured as a fourth high-availability recovery policy
  • the method further includes:
  • detecting the status of at least one virtual machine in the cluster includes:
  • the timer includes detection parameters, and the detection parameters are used to characterize the detection status,
  • the detection of the status of at least one virtual machine in the cluster includes:
  • the detection parameters determine whether the previous round of detection is completed
  • the method further includes:
  • the at least one virtual machine is removed from the high availability whitelist.
  • the method further includes:
  • the at least one virtual machine in the error task queue When the existence time of the at least one virtual machine in the error task queue reaches a specified time, the at least one virtual machine is removed from the error task queue.
  • the method further includes:
  • the method further includes:
  • the at least one virtual machine is set to a volume damaged state.
  • the method is implemented based on OpenStack.
  • a device for realizing high availability of clustered virtual machines including:
  • the status monitoring module is used to detect the status of at least one virtual machine in the cluster
  • a recovery strategy acquisition module configured to: when the state of the at least one virtual machine is an abnormal state, acquire a high-availability recovery strategy specific to the at least one virtual machine;
  • a recovery operation execution module is configured to: perform a recovery operation on the at least one virtual machine according to the acquired high-availability recovery policy specific to the at least one virtual machine.
  • an electronic device including:
  • processors one or more processors
  • a storage device for storing one or more programs
  • the one or more processors are caused to execute the method as described above.
  • a computer-readable storage medium with executable instructions stored thereon, which when executed by a processor cause the processor to perform the method as described above.
  • a computer program product comprising a computer program that when executed by a processor implements the method as described above.
  • the recovery strategy of each virtual machine can be configured, and different virtual machines can be differentially restored according to the recovery strategies specific to different virtual machines, which can greatly improve the high availability of cluster virtual machines and ensure virtual machines. Normal operation of on-board applications.
  • Figure 1 is a diagram illustrating high availability of clustered virtual machines according to some exemplary embodiments of the present disclosure. Flowchart of the method used.
  • FIG. 2 schematically illustrates a physical volume detection process in a method according to an embodiment of the present disclosure.
  • Figure 3 schematically shows a schematic diagram of communication between a computer and a virtual machine.
  • Figure 4 is a flowchart of recovery operations in a method according to some exemplary embodiments of the present disclosure.
  • Figure 5 is a schematic diagram of high-availability pipeline communications according to some exemplary embodiments of the present disclosure.
  • Figure 6 is a structural block diagram of an apparatus for realizing high availability of clustered virtual machines according to an exemplary embodiment of the present disclosure.
  • FIG. 7 schematically shows a structural block diagram of an electronic device suitable for the above method according to an exemplary embodiment of the present disclosure.
  • High Availability (English expression: High Availability, HA for short) is a technical term in distributed systems. It usually means reducing the time when the system cannot provide services through design. For example, assuming that the system can always provide services, you can It is said that the availability of the system is 100%. If the system cannot provide services for every 100 time units it runs, it can be said that the availability of the system is 99%.
  • high availability can mean: There is a problem with related resources that prevents the virtual machine from running normally. The system will restore the virtual machine and make it normal.
  • a hardware failure occurs on a physical computing node (such as disk damage, CPU or memory failure causing downtime, physical network fault and power failure)
  • the node is automatically shut down, and the virtual machine on the node is restarted on other healthy computing nodes in the cluster.
  • High availability of virtual machines mainly includes two major parts: anomaly detection and fault recovery.
  • anomaly detection of virtual machines often relies on human judgment, or detection through some simple scripts and commands, which is inefficient, cannot detect faults in a timely and accurate manner, and is not timely.
  • most technical research on fault recovery at home and abroad focuses on ensuring the host's High availability, such as using software such as pacemaker to achieve high availability of the host, and promptly migrating the virtual machines on the host to a healthy host when the host fails.
  • these technologies ignore a situation where the virtual machine may have problems and need to be rebuilt even when the host is healthy. How to accurately determine the health status of a virtual machine and rebuild a faulty virtual machine without affecting the host machine is one of the important issues focused on by the technical solutions provided by the embodiments of the present disclosure.
  • OpenStack is a project for cloud platform management. It is an open source project that currently provides software for the construction and management of public clouds, private clouds and hybrid clouds. The first priority of the Openstack project is to simplify the deployment process of the cloud and bring good scalability to it. OpenStack supports KVM, Xen, Lvc, Docker and other virtual machine software or containers.
  • OpenStack can manage data center resources and simplify resource allocation. It manages three parts of resources, namely: Computing resources: OpenStack can plan and manage a large number of virtual machines, allowing enterprises or service providers to provide computing resources on demand; developers can access computing resources through APIs to create cloud applications, and administrators and Users can access these resources through the web.
  • OpenStack can provide configurable object storage or block storage functions according to user needs;
  • Network resources There is a lot of configuration work in the data center, such as Servers, network devices, storage devices, and security devices all need to be configured, and they will be divided into more virtual devices or virtual networks, which will lead to an explosion in the number of IP addresses, routing configurations, and security rules.
  • OpenStack provides plug-in, scalable, API-driven network and IP management functions.
  • OpenStack's computing service Nova has not yet had a complete virtual machine high-availability function.
  • the high availability of user business systems relies more on the applications themselves distributed in the cluster. That is, applications running on the public cloud have their own clusters and load balancing systems, which can tolerate physical problems to a certain extent.
  • the virtual machine caused by the downtime of the computing node is unavailable, and the load on the failed computing node can be automatically transferred through the load balancing system.
  • the number of OpenStack private cloud users continues to increase. Since the non-interruptible operation of traditional centralized application systems almost completely relies on the high availability of servers, in private cloud environments , the high availability of virtual machines for OpenStack Nova services has become extremely urgent.
  • embodiments of the present disclosure provide a method for realizing high availability of cluster virtual machines, including: detecting the status of at least one virtual machine in the cluster; when the status of the at least one virtual machine is an abnormal status when the at least one virtual machine is specific to the high-availability recovery strategy; and based on the obtained high-availability recovery strategy specific to the at least one virtual machine, perform a recovery operation on the at least one virtual machine.
  • the recovery strategy of each virtual machine can be configured, and different virtual machines can be differentially restored according to the recovery strategies specific to different virtual machines, which can greatly improve the high availability of cluster virtual machines and ensure virtual machines. Normal operation of on-board applications.
  • FIG. 1 is a flowchart of a method for implementing high availability of clustered virtual machines according to some exemplary embodiments of the present disclosure.
  • the method may include operations S110 ⁇ S130.
  • detecting the status of at least one virtual machine in the cluster includes: using a timer to regularly detect the status of at least one virtual machine in the cluster according to a predetermined time interval.
  • a timer can be used to trigger the detection of OpenStack cluster virtual machines at regular intervals, and the timer can be a crontable. Crontable is a non-blocking timing trigger. When the timer is triggered, it will not detect whether the previous round of triggering has been completed. If one or more previous triggers have not been completed, multiple rounds of virtual machine exception detection will occur asynchronously. .
  • the timer includes a detection parameter, and the detection parameter Used to characterize the detection status.
  • the status of at least one virtual machine in the detection cluster includes: determining whether the previous round of detection is completed based on the detection parameters; when the previous round of detection is not completed, suspending the execution of this round of detection; when the previous round of detection is not completed; When one round of testing is completed, the current round of testing begins.
  • a detection parameter is set to characterize the detection status (that is, whether the detection is completed). The parameter status will be checked when the detection starts. If the detection When this parameter is in the detection state, it means that the previous round of detection has not been completed, then this round of detection will be stopped. Otherwise, a new round of detection will be started, and this parameter will be set to the detection state. It will not be set to the idle state until this round of detection is completed. .
  • the high-availability recovery strategy includes at least one of a first high-availability recovery strategy and a second high-availability recovery strategy, the first high-availability recovery strategy is a restart/startup strategy, and the second high-availability recovery strategy for reconstruction strategies.
  • restart/startup policy represents a restart or startup strategy.
  • start/startup policy represents a restart or startup strategy.
  • startup strategy When the virtual machine to be restored is in a shutdown state, it represents a startup strategy; when the virtual machine to be restored is in a running state, it represents a restart strategy.
  • OpenStack can provide operations at the virtual machine level, that is, starting, shutting down, creating, and deleting virtual machines. Based on the above operations, the current virtual machine has at least two recovery operations: starting/restarting the virtual machine, deleting the original virtual machine and rebuilding the virtual machine. machine.
  • high-availability recovery strategies can be configured for different virtual machines. For example, high-availability recovery strategies can include the following four types of strategies: IGNORE: ignore; RESTART: only restart (non-shutdown state)/start (shutdown state); REBUILD: only rebuild; ALL: restart/start first, and then rebuild after failure .
  • operation S130 perform a recovery operation on the at least one virtual machine according to the obtained high-availability recovery policy specific to the at least one virtual machine.
  • performing the recovery operation on the at least one virtual machine includes: detecting the at least one virtual machine The status of the host machine where the virtual machine is located; detecting the status of the physical volume where the at least one virtual machine is located; and when the status of the host machine and the physical volume are both normal, restart or start the at least one virtual machine.
  • performing a recovery operation on the at least one virtual machine includes: detecting the at least one virtual machine The status of the physical volume where the at least one virtual machine is located; detecting the status of the host machine where the at least one virtual machine is located; when the status of the host machine and the physical volume are normal, rebuilding on the host machine where the at least one virtual machine is located the at least one virtual machine.
  • the high-availability recovery strategy also includes a third high-availability recovery strategy.
  • the at least one virtual machine-specific high-availability recovery strategy is the third high-availability recovery strategy
  • the at least one virtual machine-specific high-availability recovery strategy is The recovery operation performed by a virtual machine includes: detecting the status of the host machine where the at least one virtual machine is located; detecting the status of the physical volume where the at least one virtual machine is located; when the status of the host machine and the physical volume are both normal when restarting or starting the at least one virtual machine; when restarting or starting the at least one virtual machine fails, rebuilding the at least one virtual machine on the host machine where the at least one virtual machine is located.
  • performing a recovery operation on the at least one virtual machine further includes: determining whether the at least one virtual machine Whether the status of the machine has returned to normal; when it is determined that the status of the at least one virtual machine has not returned to normal, rebuild the at least one virtual machine on other host machines, where the other host machines are located in the same location as the at least one virtual machine.
  • the hosts are in the same availability domain.
  • performing a recovery operation on the at least one virtual machine further includes: when the at least one virtual machine When the status of the host machine where the virtual machine is located is abnormal, the at least one virtual machine is rebuilt on other host machines, wherein the other host machines and the host machine where the at least one virtual machine is located are in the same available domain.
  • the at least one virtual machine is rebuilt on other hosts.
  • the virtual machine includes: sequentially traversing other hosts in the same available domain as the host where the at least one virtual machine is located, until the at least one virtual machine is successfully rebuilt on one of the other hosts in the same available domain. A virtual machine.
  • business virtual machines uniformly use OpenStack volumes as the startup source of the virtual machine system.
  • the startup volume may be damaged in the event of abnormal power outage, so the detection of the virtual machine volume must be prioritized.
  • FIG. 2 schematically illustrates a physical volume detection process in a method according to an embodiment of the present disclosure.
  • the physical volume detection process may include the following operations.
  • the physical volume can be managed using lvm or Huawei OceanStorage storage.
  • a branch corresponding to the storage mode of the physical volume is selectively entered.
  • the physical volume is managed using lvm
  • the physical volume is managed using Huawei OceanStorage
  • operation S270 it is determined that the physical volume is abnormal (unusable), and the high-availability recovery of the current virtual machine is stopped.
  • operation S291 it is determined whether the Lun storage state is normal.
  • the physical volume if and only when all physical volumes of the virtual machine are available, the physical volume is determined to be in a normal state and subsequent detection is performed; if it is detected that a physical volume is unavailable, the physical volume is The virtual machine is placed in a physical volume exception state.
  • the method may further include the step of determining whether the virtual machine is running normally. For example, you can determine whether the virtual machine is running normally by using the computer to communicate with the virtual machine.
  • Figure 3 schematically shows a schematic diagram of communication between a computer and a virtual machine.
  • the computer can communicate with the GRPC interface of the virtual machine's built-in service to detect whether the service in the virtual machine is running normally, thereby determining whether the virtual machine is running normally.
  • the Protobuf data format standard can be used to exchange data between the local computer and the virtual machine.
  • the local computer may include a GRPC client, and the GRPC client may specify the sending data and the calling interface.
  • the GRPC client may send "I am highly available" to the virtual machine.
  • the virtual machine may include a GRPC server, and the GRPC server may receive a client request to call a specified interface. For example, it may send "good for high availability" to the client.
  • the client can receive server data.
  • the method further includes: when the status of the physical volume where the at least one virtual machine is located is abnormal, setting the at least one virtual machine to the volume abnormal status; when the at least one virtual machine When the status of the physical volume where the virtual machine is located returns from abnormal to normal, perform the recovery operation again on at least one virtual machine.
  • the method further includes: when the recovery operation is successfully performed again on the at least one virtual machine, removing the volume abnormality status of the at least one virtual machine; and when the recovery operation is performed again on the at least one virtual machine.
  • the recovery operation fails, the at least one virtual machine is set to a volume damaged state.
  • performing a recovery operation on the at least one virtual machine further includes: determining whether the at least one virtual machine Whether the status of the machine has returned to normal; based on the results of the above judgment, the high-availability task queue is updated.
  • the high availability queue may include at least the following three queues: The running task queue, the error task queue and the ignored task queue.
  • the running task queue includes the virtual machine queue that is currently undergoing or waiting for high-availability recovery operations
  • the error task queue includes the virtual machine queue that has failed high-availability recovery
  • the ignored task queue includes The high availability policy is configured as IGNORE's virtual machine queue.
  • the high-availability task queue includes an error task queue
  • the error task queue includes a virtual machine queue for which high-availability recovery failed
  • the high-availability task queue is updated based on the result of the above judgment, specifically including: : When the status of the at least one virtual machine does not return to normal, add the at least one virtual machine to the error task queue.
  • the high-availability task queue also includes a running task queue.
  • the running task queue includes a virtual machine queue that is currently running or waiting to run a high-availability recovery operation. According to the result of the above judgment, update The high availability task queue specifically includes: when the state of the at least one virtual machine returns to normal, removing the at least one virtual machine from the running task queue.
  • the high-availability recovery strategy further includes a fourth high-availability recovery strategy, and the fourth high-availability recovery strategy is an ignore strategy.
  • the method further includes: when the at least one virtual machine-specific When the high-availability recovery policy is the fourth high-availability recovery policy, no recovery operation is performed on at least one virtual machine.
  • the high-availability task queue also includes an ignored task queue
  • the ignored task queue includes a virtual machine queue whose exclusive high-availability recovery policy is configured as the fourth high-availability recovery policy.
  • the method further includes: determining whether the at least one virtual machine is in at least one of a running task queue, an error task queue, and an ignored task queue. That is, the virtual machine information of the current queue can be queried in advance to determine whether the currently detected virtual machine is already in the high-availability task queue, so as to ensure that there are no duplications of virtual machines in the queue. In this way, a virtual machine can be prevented from being repeatedly added to the high availability queue, thereby reducing unnecessary detection processes.
  • FIG. 5 is a schematic diagram of high-availability pipeline communications according to some exemplary embodiments of the present disclosure.
  • the detected virtual machine is sent to other coroutines through pipelines for consumption.
  • messages from the abnormal virtual machines are consumed.
  • the method further includes: when there are other applications in the cluster When operating the at least one virtual machine, add the at least one virtual machine to a high availability whitelist, wherein the virtual machines in the high availability whitelist are temporarily configured as the fourth high availability recovery strategy; and when other applications in the cluster When the operation on the at least one virtual machine is completed, the at least one virtual machine is removed from the high availability whitelist.
  • an additional high availability whitelist function is added.
  • the virtual machine data can be added to the high-availability whitelist interface until the operation is completed and a removal message is sent to the high-availability whitelist (after shutting down the virtual machine and starting it During the entire process, the whitelist obtains the temporary IGNORE configuration policy.
  • the pipeline may include a Join Task pipeline, a Run Task pipeline, and a Leave Task pipeline.
  • the Join Task pipeline is used to receive abnormal virtual machine messages
  • the Run Task pipeline is used to receive virtual machine messages for recovery operations
  • the Leave Task pipeline is used to receive abnormal virtual machine messages.
  • the pipe is used to receive virtual machine messages after the recovery operation is completed. Send messages to the Join Task pipeline through virtual machine anomaly detection.
  • the Join Task pipeline can receive the message, determine whether the virtual machine in the message is running a task, an error task, or an ignored task, and add the running task and send it to the Run Task pipeline.
  • the Run Task pipeline can receive messages, perform high-availability recovery operations, and send the operation results to the Leave Task pipeline.
  • the Leave Task pipeline receives the message and updates the task record based on the recovery results.
  • the task record includes recorded running tasks, error tasks and ignored tasks.
  • Running tasks include abnormal virtual machine tasks to be processed, error tasks include tasks with high availability recovery failure, and ignored tasks include tasks with a high availability policy of IGNORE.
  • the method further includes: when the existence time of the at least one virtual machine in the error task queue reaches a specified time, removing the at least one virtual machine from the error task queue.
  • Figure 4 is a flowchart of recovery operations in a method according to some exemplary embodiments of the present disclosure.
  • the recovery operation in the method may include at least the following processes:
  • Restart/start-up process The host connectivity and boot volume of the virtual machine will first be checked to see if they are available. If both are available, start or restart operations will be performed, and subsequent operations will be performed based on the recovery results.
  • Reconstruction process I First, rebuild on the original host, determine whether the high availability strategy and restart/start are restored successfully, and then check whether the physical volume and host are available. If both are available, they will be deleted on the current host. The original virtual machine, and then use the physical volume, port, and quota of the original virtual machine to recreate the virtual machine, and then determine whether to perform subsequent operations based on the recovery results.
  • Reconstruction process II If the reconstruction fails on the original host, or the original host is unavailable, an attempt will be made to create a virtual machine on another host in the original host's available domain, while taking into account other high-availability solutions, such as dual-machine hot backup. High availability will ensure that the reconstructed virtual machine is not on the same host machine as the original virtual machine's host or backup machine. Traverse other valid hosts in sequence to rebuild the virtual machine until the virtual machine is successfully restored on a certain host or the host traversal is completed.
  • the virtual machine service level is monitored in real time, and through the streamlining and automatic pull-up mechanism of the exporter to ensure that the interface function is consistent with the system operating status; by specifying and managing the recovery strategy of each virtual machine , perform differentiated recovery of different virtual machines, and create backup virtual machines on any host machine in the cluster, so that cluster resources can be fully utilized and the stability of business operations can be improved.
  • embodiments of the present disclosure also provide a device for realizing high availability of a cluster virtual machine.
  • the device will be described in detail below with reference to FIG. 6 .
  • Figure 6 is a structural block diagram of an apparatus for realizing high availability of clustered virtual machines according to an exemplary embodiment of the present disclosure.
  • the device 800 for realizing high availability of cluster virtual machines includes a status monitoring module 810, a recovery strategy acquisition module 820, and a recovery operation execution module 830.
  • the status monitoring module 810 is used to detect the status of at least one virtual machine in the cluster.
  • the status monitoring module 810 may be used to perform the operation S110 and its sub-operations described above, which will not be described again here.
  • the recovery policy acquisition module 820 is configured to: when the state of the at least one virtual machine is an abnormal state, acquire a high-availability recovery policy specific to the at least one virtual machine.
  • the recovery policy acquisition module 820 may be used to perform the aforementioned operation S120 and its sub-operations, which will not be described again here.
  • the recovery operation execution module 830 is configured to: perform a recovery operation on the at least one virtual machine according to the acquired high-availability recovery policy specific to the at least one virtual machine.
  • the recovery operation execution module 830 may be used to perform the aforementioned operation S130 and its sub-operations, which will not be described again here.
  • any multiple modules among the status monitoring module 810, the recovery strategy acquisition module 820, and the recovery operation execution module 830 included in the device 800 for implementing cluster virtual machine high availability can be combined and implemented in one module, Or any one of the modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module.
  • At least one of the status monitoring module 810, the recovery strategy acquisition module 820, and the recovery operation execution module 830 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), programmable logic Array (PLA), system on a chip, system on a substrate, system on a package, application specific integrated circuit (ASIC), or any other reasonable way to integrate or package circuits, such as hardware or firmware, or in software It can be implemented in any one of the three implementation methods, hardware and firmware, or in an appropriate combination of any of them.
  • at least one of the status monitoring module 810, the recovery strategy acquisition module 820, and the recovery operation execution module 830 may be at least partially implemented as a computer program module, and when the computer program module is executed, corresponding functions may be performed.
  • FIG. 7 schematically shows a structural block diagram of an electronic device suitable for the above method according to an exemplary embodiment of the present disclosure.
  • an electronic device 1000 includes a processor 1001 that can be loaded into a random access memory (RAM) 1003 according to a program stored in a read-only memory (ROM) 1002 or from a storage part 1008 program to perform various appropriate actions and processes.
  • processor 1001 may include, for example, a general-purpose microprocessor (eg, a CPU), an instruction set processor and/or an associated chipset, and/or a special-purpose microprocessor (eg, an application specific integrated circuit (ASIC)), or the like.
  • Processor 1001 may also include onboard memory for caching purposes.
  • the processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiments of the present disclosure.
  • the processor 1001, ROM 1002 and RAM 1003 are connected to each other through a bus 1004.
  • the processor 1001 performs various operations according to the method flow of the embodiment of the present disclosure by executing programs in the ROM 1002 and/or RAM 1003. It should be noted that the program may also be stored in one or more memories other than ROM 1002 and RAM 1003.
  • the processor 1001 may also perform various operations according to the method flow of embodiments of the present disclosure by executing programs stored in the one or more memories.
  • the electronic device 1000 may further include an input/output (I/O) interface 1005 that is also connected to the bus 1004 .
  • Electronic device 1000 may also include one or more of the following components connected to I/O interface 1005: an input portion 1006 including a keyboard, mouse, etc.; including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and an output section 1007 of speakers and the like; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card, a modem and the like.
  • the communication section 1009 performs communication processing via a network such as the Internet.
  • Driver 1010 is also connected to I/O interface 1005 as needed.
  • Removable media 1011 such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.
  • the present disclosure also provides a computer-readable storage medium.
  • the computer-readable storage medium may be included in the device/device/system described in the above embodiments; it may also exist independently without being assembled into the device/system. in the device/system.
  • the above computer-readable storage medium carries one or more programs. When the above one or more programs are executed, actual Now methods according to embodiments of the present disclosure.
  • the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, but is not limited to, portable computer disks, hard disks, random access memory (RAM), and read-only memory (ROM). , erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the computer-readable storage medium may include one or more memories other than ROM 1002 and/or RAM 1003 and/or ROM 1002 and RAM 1003 described above.
  • Embodiments of the present disclosure also include a computer program product including a computer program containing program code for performing the method illustrated in the flowchart.
  • the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.
  • the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices.
  • the computer program can also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 1009, and/or installed from the removable medium 1011.
  • the program code contained in the computer program can be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
  • the computer program may be downloaded and installed from the network via communication portion 1009 and/or installed from removable media 1011.
  • the computer program is executed by the processor 1001
  • the above-described functions defined in the system of the embodiment of the present disclosure are performed.
  • the systems, devices, devices, modules, units, etc. described above may be implemented by computer program modules.
  • any combination of one or more programming languages may Together, program codes for executing computer programs provided by embodiments of the present disclosure may be written.
  • these computing programs may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages.
  • Programming languages include, but are not limited to, programming languages such as Java, C++, python, "C" language or similar programming languages.
  • the program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server.
  • the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as provided by an Internet service. (business comes via Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service business comes via Internet connection
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
  • each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or may be implemented by special purpose hardware-based systems that perform the specified functions or operations. Achieved by a combination of specialized hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Hardware Redundancy (AREA)
  • Debugging And Monitoring (AREA)

Abstract

提供了一种实现集群虚拟机高可用的方法和装置。所述方法包括：检测集群中至少一个虚拟机的状态；当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。

Description

实现集群虚拟机高可用的方法和装置、设备和介质 技术领域
本公开涉及云计算技术领域，具体涉及一种实现集群虚拟机高可用的方法和装置、电子设备、计算机可读存储介质和程序产品。
背景技术
随着云计算行业的持续兴起，OpenStack被广泛应用到各行各业，逐渐成为各大公司的主流虚拟机管理软件。但是OpenStack自身并没有一套完整的虚拟机高可用解决方案。当虚拟机出现故障时，在没有人为干预的情况下，虚拟机往往不能自行恢复，从而导致在虚拟机上运行的应用不可用，影响整个云上业务。
因此，研究一种基于OpenStack实现的集群虚拟机高可用系统对于保障云上应用的高可用性有重要意义。
在本部分中公开的以上信息仅用于对本公开的发明构思的背景的理解，因此，以上信息可包含不构成现有技术的信息。
发明内容
鉴于上述技术问题中的至少一个方面，提出了一种实现集群虚拟机高可用的方法和装置、电子设备、计算机可读存储介质和程序产品。
在一个方面，提供一种实现集群虚拟机高可用的方法，包括：
检测集群中至少一个虚拟机的状态；
当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及
根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
根据一些示例性的实施例，所述高可用恢复策略包括第一高可用恢复策略和第二高可用恢复策略中的至少一种，所述第一高可用恢复策略为重启/启动策略，所述第二高可用恢复策略为重建策略。
根据一些示例性的实施例，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
检测所述至少一个虚拟机所在的宿主机的状态；
检测所述至少一个虚拟机所在的物理卷的状态；
当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机。
根据一些示例性的实施例，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
检测所述至少一个虚拟机所在的物理卷的状态；
检测所述至少一个虚拟机所在的宿主机的状态；
当所述宿主机和所述物理卷的状态均正常时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
根据一些示例性的实施例，所述高可用恢复策略还包括第三高可用恢复策略，当所述至少一个虚拟机专属的高可用恢复策略为第三高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
检测所述至少一个虚拟机所在的宿主机的状态；
检测所述至少一个虚拟机所在的物理卷的状态；
当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机；
当重启或启动所述至少一个虚拟机失败时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
根据一些示例性的实施例，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
判断所述至少一个虚拟机的状态是否恢复正常；
当确定所述至少一个虚拟机的状态未恢复正常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
根据一些示例性的实施例，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
当所述至少一个虚拟机所在的宿主机的状态异常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
根据一些示例性的实施例，所述在其他宿主机上重建所述至少一个虚拟机包括：
依次遍历与所述至少一个虚拟机所在的宿主机在同一可用域下的其他各个宿主机，直到在同一可用域下的其他各个宿主机中的一个宿主机上成功重建所述至少一个虚拟机。
根据一些示例性的实施例，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
判断所述至少一个虚拟机的状态是否恢复正常；
根据上述判断的结果，更新高可用任务队列。
根据一些示例性的实施例，所述高可用任务队列包括错误任务队列，所述错误任务队列包括高可用恢复失败的虚拟机队列，
所述根据上述判断的结果，更新高可用任务队列，具体包括：
当所述至少一个虚拟机的状态未恢复正常时，将所述至少一个虚拟机加入错误任务队列中。
根据一些示例性的实施例，所述高可用任务队列还包括运行任务队列，所述运行任务队列包括当前正在运行或等待运行高可用恢复操作的虚拟机队列，
所述根据上述判断的结果，更新高可用任务队列，具体包括：
当所述至少一个虚拟机的状态恢复正常时，将所述至少一个虚拟机从运行任务队列中移除。
根据一些示例性的实施例，所述高可用恢复策略还包括第四高可用恢复策略，所述第四高可用恢复策略为忽略策略，
所述方法还包括：
当所述至少一个虚拟机专属的高可用恢复策略为第四高可用恢复策略时，不对该至少一个虚拟机执行恢复操作。
根据一些示例性的实施例，所述高可用任务队列还包括忽略任务队列，所述忽略任务队列包括专属的高可用恢复策略被配置为第四高可用恢复策略的虚拟机队列，
在所述检测集群中至少一个虚拟机的状态之前，所述方法还包括：
判断所述至少一个虚拟机是否在运行任务队列、错误任务队列和忽略任务队列中的至少一个中。
根据一些示例性的实施例，所述检测集群中至少一个虚拟机的状态包括：
通过定时器按照预定的时间间隔定时检测集群中至少一个虚拟机的状态。
根据一些示例性的实施例，所述定时器包括检测参数，所述检测参数用于表征检测状态，
所述检测集群中至少一个虚拟机的状态包括：
根据所述检测参数，确定上一轮检测是否完成；
当上一轮检测未完成时，暂停执行本轮检测；
当上一轮检测已完成时，开始执行本轮检测。
根据一些示例性的实施例，所述方法还包括：
当集群中存在其他应用操作所述至少一个虚拟机时，将所述至少一个虚拟机加入高可用白名单中，其中，高可用白名单中的虚拟机被暂时配置为第四高可用恢复策略；以及
当集群中其他应用完成对所述至少一个虚拟机的操作时，将所述至少一个虚拟机从高可用白名单中移除。
根据一些示例性的实施例，所述方法还包括：
当所述至少一个虚拟机在错误任务队列中存在的时间达到指定时间时，将所述至少一个虚拟机从错误任务队列中移除。
根据一些示例性的实施例，所述方法还包括：
当所述至少一个虚拟机所在的物理卷的状态为异常时，将所述至少一个虚拟机置为卷异常状态；
当所述至少一个虚拟机所在的物理卷的状态从异常恢复为正常时，对该至少一个虚拟机再次执行恢复操作。
根据一些示例性的实施例，所述方法还包括：
当对该至少一个虚拟机再次执行恢复操作成功时，移除所述至少一个虚拟机的卷异常状态；
当对该至少一个虚拟机再次执行恢复操作失败时，将所述至少一个虚拟机置为卷损坏状态。
根据一些示例性的实施例，所述方法是基于OpenStack实现的。
在另一方面，提供一种实现集群虚拟机高可用的装置，包括：
状态监测模块，用于检测集群中至少一个虚拟机的状态；
恢复策略获取模块，用于：当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及
恢复操作执行模块，用于：根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
在又另一方面，提供一种电子设备，包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序，
其中，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器执行如上所述的方法。
在再一方面，提供一种计算机可读存储介质，其上存储有可执行指令，该指令被处理器执行时使处理器执行如上所述的方法。
在仍另一方面，提供一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现如上所述的方法。
在根据本公开的实施例的方法中，可以配置每一个虚拟机的恢复策略，根据不同虚拟机专属的恢复策略对不同虚拟机进行差异化恢复，能够大大提升集群虚拟机的高可用性，保障虚拟机上应用的正常运行。
附图说明
为了更好地理解本公开，将根据以下附图对本公开进行详细描述：
图1是根据本公开的一些示例性实施例的实现集群虚拟机高可 用的方法的流程图。
图2示意性示出了根据本公开的实施例的方法中的物理卷的检测过程。
图3示意性示出了计算机与虚拟机进行通信的示意图。
图4是根据本公开的一些示例性实施例的方法中的恢复操作的流程图。
图5是根据本公开的一些示例性实施例的高可用管道通信的示意图。
图6是根据本公开的示例性实施例的实现集群虚拟机高可用的装置的结构框图。
图7示意性示出了根据本公开的示例性实施例的适于上述方法的电子设备的结构框图。
具体实施方式
下面将详细描述本公开的具体实施例，应当注意，这里描述的实施例只用于举例说明，并不用于限制本公开。在以下描述中，为了提供对本公开的透彻理解，阐述了大量特定细节。然而，对于本领域普通技术人员显而易见的是：不必采用这些特定细节来实行本公开。在其他实例中，为了避免混淆本公开，未具体描述公知的结构、材料或方法。
在整个说明书中，对“一个实施例”、“实施例”、“一个示例”或“示例”的提及意味着：结合该实施例或示例描述的特定特征、结构或特性被包含在本公开至少一个实施例中。因此，在整个说明书的各个地方出现的短语“在一个实施例中”、“在实施例中”、“一个示例”或“示例”不一定都指同一实施例或示例。此外，可以以任何适当的组合和/或子组合将特定的特征、结构或特性组合在一个或多个实施例或示例中。此外，本领域普通技术人员应当理解，这里使用的术语“和/或”包括一个或多个相关列出的项目的任何和所有组合。
在此使用的术语仅仅是为了描述具体实施例，而并非意在限制本公开。在此使用的术语“包括”、“包含”等表明了所述特征、步骤、操 作和/或部件的存在，但是并不排除存在或添加一个或多个其他特征、步骤、操作或部件。
在此使用的所有术语(包括技术和科学术语)具有本领域技术人员通常所理解的含义，除非另外定义。应注意，这里使用的术语应解释为具有与本说明书的上下文相一致的含义，而不应以理想化或过于刻板的方式来解释。
在本文中，“高可用(英文表述为High Availability，简称为HA”是分布式系统中的技术术语，它通常表示通过设计减少系统不能提供服务的时间。例如，假设系统一直能够提供服务，可以说系统的可用性是100％。如果系统每运行100个时间单位，会有1个时间单位无法提供服务，可以说系统的可用性是99％。在集群虚拟机领域，高可用可以表示：虚拟机的相关资源发生了问题，使得虚拟机不能正常运行，系统会恢复该虚拟机使得它变为正常。例如，在物理计算节点发生硬件故障时(例如磁盘损坏、CPU或内存故障导致宕机、物理网络故障和电源故障)，自动将该节点关闭，该节点上的虚拟机在集群中其它健康的计算节点上重启。
发明人经研究发现，虚拟机的健康状况是保证云上业务正常运行的关键因素，业务环境中的运行异常或者故障会对业务的正常运行造成不可控的影响，严重时甚至会导致整个业务功能的瘫痪。因此，使业务运行系统具备一定的异常应对能力，将是提高整个业务系统稳定性的重要措施之一。高可用是保障系统在突发异常时依然能够对外提供正常功能的一种常见解决方案，在目前的实际业务中使用了集群化、双机热备等方案来保障业务核心服务所在虚拟机的高可用，但以上两种方案的异常处理过程都有相应的弊端，如集群部署时某一节点的异常会增加其它节点的运行负载，双机热备无法应对主机与备机同时故障的情况。
虚拟机的高可用主要包含异常检测和故障恢复两大块。目前，虚拟机的异常检测往往依靠人为判定，或者通过一些简单的脚本、命令进行检测，效率较低，且不能做到及时准确发现故障，时效性不高。另一方面，国内外针对故障恢复的技术研究大多集中于保障宿主机的 高可用性，比如利用pacemaker等软件实现宿主机的高可用，在宿主机出现故障时及时将宿主机上的虚拟机迁移至健康宿主机上。但是这些技术忽略了一种情况，即宿主机健康时虚拟机也有可能出现问题而需要重建。如何准确判断出虚拟机的健康状况，并在不影响宿主机的情况下对故障虚拟机进行重建，是本公开实施例提供的技术方案关注的重要课题之一。
在描述本公开的实施例提供的方法之前，对OpenStack系统做一个简要的介绍。需要说明的是，在本部分中，为了本公开的实施例提供的方法得到更好的理解，结合OpenStack系统对本公开的实施例提供的方法进行了详细描述。但是，这样的描述不应视为对本公开的实施例的限制，也就是说，在本公开的实施例中，所述方法可以是基于OpenStack系统实现的，但是，本公开的实施例不局限于此，所述方法也可以基于其他虚拟机管理系统实现。
OpenStack是一个用于云平台管理的项目，目前为公有云、私有云和混合云的建设与管理提供软件的开源项目。Openstack项目的首要任务是简化云的部署过程并为其带来良好的可扩展性。OpenStack支持KVM、Xen、Lvc、Docker等虚拟机软件或容器。
OpenStack的主要目标是管理数据中心的资源，简化资源分配。它管理三部分资源，分别是：计算资源：OpenStack可以规划并管理大量虚拟机，从而允许企业或服务提供商按需提供计算资源；开发者可以通过API访问计算资源从而创建云应用，管理员与用户则可以通过web访问这些资源。存储资源：OpenStack可以为云服务或云应用提供所需的对象及块存储资源。因为对性能及价格有需求，很多组织已经不能满足于传统的企业级存储技术，因此OpenStack可以根据用户需要提供可配置的对象存储或块存储功能；网络资源：数据中心存在大量的配置工作，例如服务器、网络设备、存储设备、安全设备均需要配置，而它们还将被划分成更多的虚拟设备或虚拟网络，这会导致IP地址的数量、路由配置、安全规则爆炸式增长，鉴于此，OpenStack提供了插件式、可扩展、API驱动型的网络及IP管理功能。
发明人经研究发现，由于OpenStack的初衷是面向公有云，因此OpenStack在“基因”里便缺乏虚拟机高可用设计，这也是到目前为止OpenStack的计算服务Nova一直没有完善的虚拟机高可用功能的原因。在公有云环境中，用户业务系统的高可用性更多地依赖于分布在集群中的应用程序本身，即公有云上运行的应用程序有自己的集群和负载均衡系统，能在一定程度上容忍物理计算节点宕机所带来的虚拟机不可用，并能通过负载均衡系统实现故障计算节点上的负载自动转移。随着越来越多传统IT架构下的用户转入云计算，OpenStack的私有云用户不断增加，由于传统集中式应用系统的非中断运行几乎完全依赖于服务器的高可用性，因此在私有云环境下，OpenStack Nova服务的虚拟机高可用性变得极为迫切。
针对上述问题的至少一个方面，本公开的实施例提供了一种实现集群虚拟机高可用的方法，包括：检测集群中至少一个虚拟机的状态；当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
在根据本公开的实施例的方法中，可以配置每一个虚拟机的恢复策略，根据不同虚拟机专属的恢复策略对不同虚拟机进行差异化恢复，能够大大提升集群虚拟机的高可用性，保障虚拟机上应用的正常运行。
图1是根据本公开的一些示例性实施例的实现集群虚拟机高可用的方法的流程图。参照图1，所述方法可以包括操作S110～S130。
在操作S110中，检测集群中至少一个虚拟机的状态。
在本公开的实施例中，所述检测集群中至少一个虚拟机的状态包括：通过定时器按照预定的时间间隔定时检测集群中至少一个虚拟机的状态。
例如，可以由定时器每隔一段时间触发OpenStack集群虚拟机的检测，定时器可以采用crontable。crontable是一种非阻塞式的定时触发器，该定时器触发时不会检测上一轮触发是否完成，如果前一次或多次触发尚未完成，则会出现多轮虚拟机异常检测异步进行的情况。
在本公开的实施例中，所述定时器包括检测参数，所述检测参数 用于表征检测状态，所述检测集群中至少一个虚拟机的状态包括：根据所述检测参数，确定上一轮检测是否完成；当上一轮检测未完成时，暂停执行本轮检测；当上一轮检测已完成时，开始执行本轮检测。
也就是说，在本公开的实施例中，针对crontable非阻塞特点导致的问题，设置了一个用于表征检测状态(即检测是否完成)的检测参数，检测开始时会检查该参数状态，如果检测到该参数为检测状态，表示上一轮检测还未完成，则停止本轮检测，否则就开始新一轮检测，并将该参数设为检测状态，直到该轮检测完成时才设置为闲置状态。
在操作S120中，当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略。
例如，所述高可用恢复策略包括第一高可用恢复策略和第二高可用恢复策略中的至少一种，所述第一高可用恢复策略为重启/启动策略，所述第二高可用恢复策略为重建策略。
需要说明的是，表述“重启/启动策略”表示重启或启动策略，在待恢复的虚拟机处于关闭状态时，其表示启动策略；在待恢复的虚拟机处于运行状态时，其表示重启策略。
例如，OpenStack能提供虚拟机层面的操作，即启动、关闭、创建、删除虚拟机操作，基于以上操作，当前虚拟机至少有两种恢复操作：启动/重启虚拟机，删除原虚拟机后重建虚拟机。根据实际业务的不同，不同虚拟机可以配置不同的高可用恢复策略。例如，高可用恢复策略可以包括以下四类策略：IGNORE：忽略；RESTART：只重启(非关闭状态)/启动(关闭状态)；REBUILD：只重建；ALL：先进行重启/启动，失败后再重建。
在本公开的实施例中，可以将“RESTART：只重启(非关闭状态)/启动(关闭状态)”描述为第一高可用恢复策略，将“只重建；ALL”描述为第二高可用恢复策略，将“ALL：先进行重启/启动，失败后再重建”描述为第三高可用恢复策略，将“IGNORE：忽略”描述为第四高可用恢复策略。
在操作S130中，根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
在本公开的实施例中，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：检测所述至少一个虚拟机所在的宿主机的状态；检测所述至少一个虚拟机所在的物理卷的状态；当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机。
在本公开的实施例中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：检测所述至少一个虚拟机所在的物理卷的状态；检测所述至少一个虚拟机所在的宿主机的状态；当所述宿主机和所述物理卷的状态均正常时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
在本公开的实施例中，所述高可用恢复策略还包括第三高可用恢复策略，当所述至少一个虚拟机专属的高可用恢复策略为第三高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：检测所述至少一个虚拟机所在的宿主机的状态；检测所述至少一个虚拟机所在的物理卷的状态；当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机；当重启或启动所述至少一个虚拟机失败时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
在本公开的实施例中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：判断所述至少一个虚拟机的状态是否恢复正常；当确定所述至少一个虚拟机的状态未恢复正常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
在本公开的实施例中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：当所述至少一个虚拟机所在的宿主机的状态异常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
在本公开的实施例中，所述在其他宿主机上重建所述至少一个虚 拟机包括：依次遍历与所述至少一个虚拟机所在的宿主机在同一可用域下的其他各个宿主机，直到在同一可用域下的其他各个宿主机中的一个宿主机上成功重建所述至少一个虚拟机。
例如，业务虚拟机统一采用OpenStack卷作为虚拟机系统启动源，启动卷在断电异常情况下有概率出现损坏，对虚拟机卷的检测要优先进行。
图2示意性示出了根据本公开的实施例的方法中的物理卷的检测过程。参照图2，所述物理卷的检测过程可以包括以下操作。
在操作S210中，获取物理卷的存储方式，例如，根据测试环境的不同，物理卷可以采用lvm的方式管理，或者采用华为OceanStorage存储。
在操作S220中，根据获取的物理卷的存储方式，选择性进入与物理卷的存储方式对应的分支。当物理卷采用lvm的方式管理时，进入图2中的左侧分支；当物理卷采用华为OceanStorage存储的方式管理时，进入图2中的右侧分支。
在操作S230中，当物理卷采用lvm的方式管理时，通过ssh连接到物理卷所在的物理机。
在操作S240中，判断通过ssh与物理卷所在的物理机是否连通。
当通过ssh与物理卷所在的物理机连通时，进行操作S250。在操作S250中，判断返回信息中是否包含指定物理卷。
当返回信息中包含指定物理卷时，进行操作S260。在操作S260中，确定物理卷正常，可以进行高可用恢复。
当通过ssh与物理卷所在的物理机未连通或返回信息中未包含指定物理卷时，进行操作S270。在操作S270中，确定物理卷异常(无法使用)，停止当前虚拟机的高可用恢复。
在操作S280中，当物理卷采用华为OceanStorage存储的方式管理时，查询DeviceManager中物理卷信息。
在操作S290中，判断是否有物理卷信息。
当有物理卷信息时，进行操作S291。在操作S291中，判断Lun存储状态是否正常。
当Lun存储状态正常时，进行上述操作S260。
当没有物理卷信息或Lun存储状态不正常时，进行上述操作S270。
在图2所示的实施例中，当且仅当虚拟机的所有物理卷都可用时，才确定物理卷为正常状态，才会进行后续检测；如果检测到有物理卷不可用，会将该虚拟机置于物理卷异常状态。
在本公开的实施例中，在对至少一个虚拟机执行恢复操作后，所述方法还可以包括判断虚拟机是否正常运行的步骤。例如，可以通过利用计算机与虚拟机进行通信的方式，来判断虚拟机是否正常运行。图3示意性示出了计算机与虚拟机进行通信的示意图。参照图3，计算机可以与虚拟机内置服务的GRPC接口进行通信，以检测虚拟机内服务是否正常运行，从而判断虚拟机是否正常运行。例如，可以采用Protobuf数据格式标准在本地计算机和虚拟机之间交互数据。本地计算机可以包括GRPC客户端，所述GRPC客户端可以指定发送数据和指定调用接口，例如，所述GRPC客户端可以发送“我是高可用”给虚拟机。所述虚拟机可以包括GRPC服务端，所述GRPC服务端可以接收客户端请求调用指定接口，例如，可以发送“利好高可用”给客户端。所述客户端可以接收服务端数据。
在本公开的实施例中，所述方法还包括：当所述至少一个虚拟机所在的物理卷的状态为异常时，将所述至少一个虚拟机置为卷异常状态；当所述至少一个虚拟机所在的物理卷的状态从异常恢复为正常时，对该至少一个虚拟机再次执行恢复操作。
在本公开的实施例中，所述方法还包括：当对该至少一个虚拟机再次执行恢复操作成功时，移除所述至少一个虚拟机的卷异常状态；当对该至少一个虚拟机再次执行恢复操作失败时，将所述至少一个虚拟机置为卷损坏状态。
在本公开的实施例中，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：判断所述至少一个虚拟机的状态是否恢复正常；根据上述判断的结果，更新高可用任务队列。
在本公开的实施例中，高可用队列可以包括以下至少三个队列： 运行任务队列、错误任务队列和忽略任务队列，其中，运行任务队列包括当前正在进行或等待进行高可用恢复操作的虚拟机队列，错误任务队列包括高可用恢复失败的虚拟机队列，忽略任务队列包括高可用策略配置为IGNORE的虚拟机队列。
在本公开的实施例中，所述高可用任务队列包括错误任务队列，所述错误任务队列包括高可用恢复失败的虚拟机队列，所述根据上述判断的结果，更新高可用任务队列，具体包括：当所述至少一个虚拟机的状态未恢复正常时，将所述至少一个虚拟机加入错误任务队列中。
在本公开的实施例中，所述高可用任务队列还包括运行任务队列，所述运行任务队列包括当前正在运行或等待运行高可用恢复操作的虚拟机队列，所述根据上述判断的结果，更新高可用任务队列，具体包括：当所述至少一个虚拟机的状态恢复正常时，将所述至少一个虚拟机从运行任务队列中移除。
在本公开的实施例中，所述高可用恢复策略还包括第四高可用恢复策略，所述第四高可用恢复策略为忽略策略，所述方法还包括：当所述至少一个虚拟机专属的高可用恢复策略为第四高可用恢复策略时，不对该至少一个虚拟机执行恢复操作。
在本公开的实施例中，所述高可用任务队列还包括忽略任务队列，所述忽略任务队列包括专属的高可用恢复策略被配置为第四高可用恢复策略的虚拟机队列，在所述检测集群中至少一个虚拟机的状态之前，所述方法还包括：判断所述至少一个虚拟机是否在运行任务队列、错误任务队列和忽略任务队列中的至少一个中。即，可以事先对当前队列的虚拟机信息进行查询，判断当前检测的虚拟机是否已经位于高可用任务队列中，以保证队列中的虚拟机不出现重复。这样，可以防止某个虚拟机被重复添加到高可用队列中，从而可以减少不必要的检测过程。
图5是根据本公开的一些示例性实施例的高可用管道通信的示意图。参照图5，通过管道将检测到的虚拟机发送到其它协程进行消费，在运行协程中会对异常虚拟机的消息进行消费。
在本公开的实施例中，所述方法还包括：当集群中存在其他应用 操作所述至少一个虚拟机时，将所述至少一个虚拟机加入高可用白名单中，其中，高可用白名单中的虚拟机被暂时配置为第四高可用恢复策略；以及当集群中其他应用完成对所述至少一个虚拟机的操作时，将所述至少一个虚拟机从高可用白名单中移除。具体地，考虑到除了高可用外，集群中还有其它自研应用服务也会对OpenStack中的虚拟机进行操作，因此，在本公开的实施例中，额外添加了一个高可用白名单功能，当其它应用操纵虚拟机进行例如启动、重启、关闭和创建等操作时，可以向高可用白名单接口中添加虚拟机数据，直到操作完成向高可用白名单发送移除消息(关闭虚拟机启动后视为完成操作)整个过程中，白名单获得暂时的IGNORE的配置策略。
例如，参照图5，管道可以包括Join Task管道、Run Task管道、Leave Task管道，其中，Join Task管道用于接收异常虚拟机消息，Run Task管道用于接收进行恢复操作的虚拟机消息，Leave Task管道用于接收完成恢复操作后的虚拟机消息。通过虚拟机异常检测，向Join Task管道发消息。在入队列中，Join Task管道可以接收所述消息，判断消息中的虚拟机是否在运行任务、错误任务、忽略任务中，并加入运行任务和发送到Run Task管道。在运行队列中，Run Task管道可以接收消息，执行高可用恢复操作，并将操作结果发送到Leave Task管道。在出队列中，Leave Task管道接收消息，并根据恢复结果更新任务记录。所述任务记录包括记录的运行任务、错误任务和忽略任务，运行任务包括待处理的异常虚拟机任务，错误任务包括高可用恢复失败的任务，忽略任务包括高可用策略为IGNORE的任务。
在本公开的实施例中，所述方法还包括：当所述至少一个虚拟机在错误任务队列中存在的时间达到指定时间时，将所述至少一个虚拟机从错误任务队列中移除。
图4是根据本公开的一些示例性实施例的方法中的恢复操作的流程图。参照图4，所述方法中的恢复操作可以至少包括以下几个过程：
(1)从配置文件或者数据库获取宿主机的高可用策略信息，数据库配置优先级更高，根据高可用策略决定进行重启/启动(Restart/ALL) 还是重建(Rebuild/ALL)。虽然当前测试环境统一采用了物理卷方式创建虚拟机，高可用也兼容镜像方式创建的虚拟机。
(2)重启/启动过程：会先检测虚拟机的宿主机连通性和启动卷是否可用，如果均可用则进行启动或者重启操作，并根据恢复结果进行后续操作
(3)重建过程I：首先是原宿主机上的重建，判断高可用策略以及重启/启动是否恢复成功，然后会检查物理卷和宿主机是否可用，如果均可用则会在当前宿主机上先删除原有虚拟机，然后适用原虚拟机的物理卷，端口，配额重新创建虚拟机，然后根据恢复结果判断是否进行后续操作。
(4)重建过程II：在原宿主机上重建失败，或者原宿主机不可用，会在原宿主机可用域下的其它宿主机上尝试创建虚拟机，同时考虑到其它高可用方案，如双机热备的虚拟机，高可用会保证重建的虚拟机与原虚拟机的主机或者备机不在同一个宿主机上。依次遍历有效的其它宿主机进行虚拟机的重建，直到虚拟机在某一个宿主机上恢复成功或者宿主机遍历完成。
在本公开的实施例中，依靠业务环境中OpenStack对虚拟机管理的检测和支持，以及Prometheus对虚拟机内系统服务的监控，提出一套既能检测虚拟机异常又能检测虚拟机内部运行系统异常，并能根据不同异常情况灵活应对的虚拟机高可用系统。通过心跳式的访问exporter中的GRPC接口，对虚拟机服务层面实时监控，并且通过对exporter的精简与自动拉起机制保证接口功能与系统运行状态一致；通过指定并管理每一个虚拟机的恢复策略，对不同虚拟机进行差异化恢复，在集群内的任意宿主机上创建备用虚拟机，使集群资源得到充分利用，提升业务运行的稳定性。
基于上述实现集群虚拟机高可用的方法，本公开的实施例还提供了一种实现集群虚拟机高可用的装置。以下将结合图6对该装置进行详细描述。
图6是根据本公开的示例性实施例的实现集群虚拟机高可用的装置的结构框图。
如图6所示，所述实现集群虚拟机高可用的装置800包括状态监测模块810、恢复策略获取模块820和恢复操作执行模块830。
状态监测模块810用于检测集群中至少一个虚拟机的状态。在一些示例性的实施例中，状态监测模块810可以用于执行前文描述的操作S110及其子操作，在此不再赘述。
恢复策略获取模块820用于：当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略。在一些示例性的实施例中，恢复策略获取模块820可以用于执行前文描述的操作S120及其子操作，在此不再赘述。
恢复操作执行模块830用于：根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。在一些示例性的实施例中，恢复操作执行模块830可以用于执行前文描述的操作S130及其子操作，在此不再赘述。
根据本公开的实施例，所述实现集群虚拟机高可用的装置800包括的状态监测模块810、恢复策略获取模块820和恢复操作执行模块830中的任意多个模块可以合并在一个模块中实现，或者其中的任意一个模块可以被拆分成多个模块。或者，这些模块中的一个或多个模块的至少部分功能可以与其他模块的至少部分功能相结合，并在一个模块中实现。根据本公开的实施例，状态监测模块810、恢复策略获取模块820和恢复操作执行模块830中的至少一个可以至少被部分地实现为硬件电路，例如现场可编程门阵列(FPGA)、可编程逻辑阵列(PLA)、片上系统、基板上的系统、封装上的系统、专用集成电路(ASIC)，或可以通过对电路进行集成或封装的任何其他的合理方式等硬件或固件来实现，或以软件、硬件以及固件三种实现方式中任意一种或以其中任意几种的适当组合来实现。或者，状态监测模块810、恢复策略获取模块820和恢复操作执行模块830中的至少一个可以至少被部分地实现为计算机程序模块，当该计算机程序模块被运行时，可以执行相应的功能。
图7示意性示出了根据本公开的示例性实施例的适于上述方法的电子设备的结构框图。
如图7所示，根据本公开实施例的电子设备1000包括处理器1001，其可以根据存储在只读存储器(ROM)1002中的程序或者从存储部分1008加载到随机访问存储器(RAM)1003中的程序而执行各种适当的动作和处理。处理器1001例如可以包括通用微处理器(例如CPU)、指令集处理器和/或相关芯片组和/或专用微处理器(例如，专用集成电路(ASIC))等等。处理器1001还可以包括用于缓存用途的板载存储器。处理器1001可以包括用于执行根据本公开实施例的方法流程的不同动作的单一处理单元或者是多个处理单元。
在RAM 1003中，存储有电子设备1000操作所需的各种程序和数据。处理器1001、ROM 1002以及RAM 1003通过总线1004彼此相连。处理器1001通过执行ROM 1002和/或RAM 1003中的程序来执行根据本公开实施例的方法流程的各种操作。需要注意，所述程序也可以存储在除ROM 1002和RAM 1003以外的一个或多个存储器中。处理器1001也可以通过执行存储在所述一个或多个存储器中的程序来执行根据本公开实施例的方法流程的各种操作。
根据本公开的实施例，电子设备1000还可以包括输入/输出(I/O)接口1005，输入/输出(I/O)接口1005也连接至总线1004。电子设备1000还可以包括连接至I/O接口1005的以下部件中的一项或多项：包括键盘、鼠标等的输入部分1006；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分1007；包括硬盘等的存储部分1008；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分1009。通信部分1009经由诸如因特网的网络执行通信处理。驱动器1010也根据需要连接至I/O接口1005。可拆卸介质1011，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器1010上，以便于从其上读出的计算机程序根据需要被安装入存储部分1008。
本公开还提供了一种计算机可读存储介质，该计算机可读存储介质可以是上述实施例中描述的设备/装置/系统中所包含的；也可以是单独存在，而未装配入该设备/装置/系统中。上述计算机可读存储介质承载有一个或者多个程序，当上述一个或者多个程序被执行时，实 现根据本公开实施例的方法。
根据本公开的实施例，计算机可读存储介质可以是非易失性的计算机可读存储介质，例如可以包括但不限于：便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。例如，根据本公开的实施例，计算机可读存储介质可以包括上文描述的ROM 1002和/或RAM 1003和/或ROM 1002和RAM 1003以外的一个或多个存储器。
本公开的实施例还包括一种计算机程序产品，其包括计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。当计算机程序产品在计算机系统中运行时，该程序代码用于使计算机系统实现本公开实施例所提供的物品推荐方法。
在该计算机程序被处理器1001执行时执行本公开实施例的系统/装置中限定的上述功能。根据本公开的实施例，上文描述的系统、装置、模块、单元等可以通过计算机程序模块来实现。
在一种实施例中，该计算机程序可以依托于光存储器件、磁存储器件等有形存储介质。在另一种实施例中，该计算机程序也可以在网络介质上以信号的形式进行传输、分发，并通过通信部分1009被下载和安装，和/或从可拆卸介质1011被安装。该计算机程序包含的程序代码可以用任何适当的网络介质传输，包括但不限于：无线、有线等等，或者上述的任意合适的组合。
在这样的实施例中，该计算机程序可以通过通信部分1009从网络上被下载和安装，和/或从可拆卸介质1011被安装。在该计算机程序被处理器1001执行时，执行本公开实施例的系统中限定的上述功能。根据本公开的实施例，上文描述的系统、设备、装置、模块、单元等可以通过计算机程序模块来实现。
根据本公开的实施例，可以以一种或多种程序设计语言的任意组 合来编写用于执行本公开实施例提供的计算机程序的程序代码，具体地，可以利用高级过程和/或面向对象的编程语言、和/或汇编/机器语言来实施这些计算程序。程序设计语言包括但不限于诸如Java，C++，python，“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。在涉及远程计算设备的情形中，远程计算设备可以通过任意种类的网络，包括局域网(LAN)或广域网(WAN)，连接到用户计算设备，或者，可以连接到外部计算设备(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
本领域技术人员可以理解，本公开的各个实施例和/或权利要求中记载的特征可以进行多种组合或/或结合，即使这样的组合或结合没有明确记载于本公开中。特别地，在不脱离本公开精神和教导的情况下，本公开的各个实施例和/或权利要求中记载的特征可以进行多种组合和/或结合。所有这些组合和/或结合均落入本公开的范围。
以上对本公开的实施例进行了描述。但是，这些实施例仅仅是为了说明的目的，而并非为了限制本公开的范围。尽管在以上分别描述了各实施例，但是这并不意味着各个实施例中的措施不能有利地结合使用。本公开的范围由所附权利要求及其等同物限定。不脱离本公开的范围，本领域技术人员可以做出多种替代和修改，这些替代和修改 都应落在本公开的范围之内。

Claims (24)

  1. 一种实现集群虚拟机高可用的方法，其特征在于，包括：
    检测集群中至少一个虚拟机的状态；
    当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及
    根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
  2. 根据权利要求1所述的方法，其中，所述高可用恢复策略包括第一高可用恢复策略和第二高可用恢复策略中的至少一种，
    所述第一高可用恢复策略为重启/启动策略，所述第二高可用恢复策略为重建策略。
  3. 根据权利要求2所述的方法，其中，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
    检测所述至少一个虚拟机所在的宿主机的状态；
    检测所述至少一个虚拟机所在的物理卷的状态；
    当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机。
  4. 根据权利要求2或3所述的方法，其中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
    检测所述至少一个虚拟机所在的物理卷的状态；
    检测所述至少一个虚拟机所在的宿主机的状态；
    当所述宿主机和所述物理卷的状态均正常时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
  5. 根据权利要求4所述的方法，其中，所述高可用恢复策略还包括第三高可用恢复策略，当所述至少一个虚拟机专属的高可用恢复策略为第三高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作包括：
    检测所述至少一个虚拟机所在的宿主机的状态；
    检测所述至少一个虚拟机所在的物理卷的状态；
    当所述宿主机和所述物理卷的状态均正常时，重启或启动所述至少一个虚拟机；
    当重启或启动所述至少一个虚拟机失败时，在所述至少一个虚拟机所在的宿主机上重建所述至少一个虚拟机。
  6. 根据权利要求4所述的方法，其中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
    判断所述至少一个虚拟机的状态是否恢复正常；
    当确定所述至少一个虚拟机的状态未恢复正常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
  7. 根据权利要求4所述的方法，其中，当所述至少一个虚拟机专属的高可用恢复策略为第二高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
    当所述至少一个虚拟机所在的宿主机的状态异常时，在其他宿主机上重建所述至少一个虚拟机，其中，所述其他宿主机与所述至少一个虚拟机所在的宿主机在同一可用域下。
  8. 根据权利要求7所述的方法，其中，所述在其他宿主机上重建所述至少一个虚拟机包括：
    依次遍历与所述至少一个虚拟机所在的宿主机在同一可用域下 的其他各个宿主机，直到在同一可用域下的其他各个宿主机中的一个宿主机上成功重建所述至少一个虚拟机。
  9. 根据权利要求4所述的方法，其中，当所述至少一个虚拟机专属的高可用恢复策略为第一高可用恢复策略时，所述对该至少一个虚拟机执行恢复操作还包括：
    判断所述至少一个虚拟机的状态是否恢复正常；
    根据上述判断的结果，更新高可用任务队列。
  10. 根据权利要求9所述的方法，其中，所述高可用任务队列包括错误任务队列，所述错误任务队列包括高可用恢复失败的虚拟机队列，
    所述根据上述判断的结果，更新高可用任务队列，具体包括：
    当所述至少一个虚拟机的状态未恢复正常时，将所述至少一个虚拟机加入错误任务队列中。
  11. 根据权利要求10所述的方法，其中，所述高可用任务队列还包括运行任务队列，所述运行任务队列包括当前正在运行或等待运行高可用恢复操作的虚拟机队列，
    所述根据上述判断的结果，更新高可用任务队列，具体包括：
    当所述至少一个虚拟机的状态恢复正常时，将所述至少一个虚拟机从运行任务队列中移除。
  12. 根据权利要求11所述的方法，其中，所述高可用恢复策略还包括第四高可用恢复策略，所述第四高可用恢复策略为忽略策略，
    所述方法还包括：
    当所述至少一个虚拟机专属的高可用恢复策略为第四高可用恢复策略时，不对该至少一个虚拟机执行恢复操作。
  13. 根据权利要求12所述的方法，其中，所述高可用任务队列还 包括忽略任务队列，所述忽略任务队列包括专属的高可用恢复策略被配置为第四高可用恢复策略的虚拟机队列，
    在所述检测集群中至少一个虚拟机的状态之前，所述方法还包括：
    判断所述至少一个虚拟机是否在运行任务队列、错误任务队列和忽略任务队列中的至少一个中。
  14. 根据权利要求1-3中任一项所述的方法，其中，所述检测集群中至少一个虚拟机的状态包括：
    通过定时器按照预定的时间间隔定时检测集群中至少一个虚拟机的状态。
  15. 根据权利要求14所述的方法，其中，所述定时器包括检测参数，所述检测参数用于表征检测状态，
    所述检测集群中至少一个虚拟机的状态包括：
    根据所述检测参数，确定上一轮检测是否完成；
    当上一轮检测未完成时，暂停执行本轮检测；
    当上一轮检测已完成时，开始执行本轮检测。
  16. 根据权利要求1-3中任一项所述的方法，其中，所述方法还包括：
    当集群中存在其他应用操作所述至少一个虚拟机时，将所述至少一个虚拟机加入高可用白名单中，其中，高可用白名单中的虚拟机被暂时配置为第四高可用恢复策略；以及
    当集群中其他应用完成对所述至少一个虚拟机的操作时，将所述至少一个虚拟机从高可用白名单中移除。
  17. 根据权利要求1-3中任一项所述的方法，其中，所述方法还包括：
    当所述至少一个虚拟机在错误任务队列中存在的时间达到指定时间时，将所述至少一个虚拟机从错误任务队列中移除。
  18. 根据权利要求1-3中任一项所述的方法，其中，所述方法还包括：
    当所述至少一个虚拟机所在的物理卷的状态为异常时，将所述至少一个虚拟机置为卷异常状态；
    当所述至少一个虚拟机所在的物理卷的状态从异常恢复为正常时，对该至少一个虚拟机再次执行恢复操作。
  19. 根据权利要求18所述的方法，其中，所述方法还包括：
    当对该至少一个虚拟机再次执行恢复操作成功时，移除所述至少一个虚拟机的卷异常状态；
    当对该至少一个虚拟机再次执行恢复操作失败时，将所述至少一个虚拟机置为卷损坏状态。
  20. 根据权利要求1-3中任一项所述的方法，其中，所述方法是基于OpenStack实现的。
  21. 一种实现集群虚拟机高可用的装置，其特征在于，包括：
    状态监测模块，用于检测集群中至少一个虚拟机的状态；
    恢复策略获取模块，用于：当所述至少一个虚拟机的状态为异常状态时，获取该至少一个虚拟机专属的高可用恢复策略；以及
    恢复操作执行模块，用于：根据获取的该至少一个虚拟机专属的高可用恢复策略，对该至少一个虚拟机执行恢复操作。
  22. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个程序，
    其中，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器执行根据权利要求1～20中任一项所述的方法。
  23. 一种计算机可读存储介质，其上存储有可执行指令，该指令被处理器执行时使处理器执行根据权利要求1～20中任一项所述的方法。
  24. 一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现根据权利要求1～20中任一项所述的方法。
PCT/CN2023/078674 2022-03-28 2023-02-28 实现集群虚拟机高可用的方法和装置、设备和介质 Ceased WO2023185355A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210316175 2022-03-28
CN202210316175.8 2022-03-28
CN202210511479.XA CN114691304B (zh) 2022-03-28 2022-05-10 实现集群虚拟机高可用的方法和装置、设备和介质
CN202210511479.X 2022-05-10

Publications (1)

Publication Number Publication Date
WO2023185355A1 true WO2023185355A1 (zh) 2023-10-05

Family

ID=85776071

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/078674 Ceased WO2023185355A1 (zh) 2022-03-28 2023-02-28 实现集群虚拟机高可用的方法和装置、设备和介质

Country Status (2)

Country Link
EP (1) EP4254191A1 (zh)
WO (1) WO2023185355A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104253860A (zh) * 2014-09-11 2014-12-31 武汉噢易云计算有限公司 一种基于共享存储消息队列的虚拟机高可用实现方法
CN108108255A (zh) * 2016-11-25 2018-06-01 中兴通讯股份有限公司 虚拟机故障的检测和恢复方法及装置
US20180165166A1 (en) * 2016-12-14 2018-06-14 Nutanix, Inc. Maintaining high availability during n-node failover
CN111880906A (zh) * 2020-08-03 2020-11-03 广东省华南技术转移中心有限公司 虚拟机高可用性管理方法、系统以及存储介质
CN114691304A (zh) * 2022-03-28 2022-07-01 同方威视技术股份有限公司 实现集群虚拟机高可用的方法和装置、设备和介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106559441B (zh) * 2015-09-25 2020-09-04 华为技术有限公司 一种基于云计算服务的虚拟机监控方法、装置及系统
CN105357038B (zh) * 2015-10-26 2019-05-07 北京百度网讯科技有限公司 监控虚拟机集群的方法和系统
CN108632057A (zh) * 2017-03-17 2018-10-09 华为技术有限公司 一种云计算服务器的故障恢复方法、装置及管理系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104253860A (zh) * 2014-09-11 2014-12-31 武汉噢易云计算有限公司 一种基于共享存储消息队列的虚拟机高可用实现方法
CN108108255A (zh) * 2016-11-25 2018-06-01 中兴通讯股份有限公司 虚拟机故障的检测和恢复方法及装置
US20180165166A1 (en) * 2016-12-14 2018-06-14 Nutanix, Inc. Maintaining high availability during n-node failover
CN111880906A (zh) * 2020-08-03 2020-11-03 广东省华南技术转移中心有限公司 虚拟机高可用性管理方法、系统以及存储介质
CN114691304A (zh) * 2022-03-28 2022-07-01 同方威视技术股份有限公司 实现集群虚拟机高可用的方法和装置、设备和介质

Also Published As

Publication number Publication date
EP4254191A1 (en) 2023-10-04

Similar Documents

Publication Publication Date Title
CN104160378B (zh) 用于启用分布式计算系统中的容错功能的方法和系统
US9052935B1 (en) Systems and methods for managing affinity rules in virtual-machine environments
US8910172B2 (en) Application resource switchover systems and methods
US8910160B1 (en) Handling of virtual machine migration while performing clustering operations
US20120180070A1 (en) Single point, scalable data synchronization for management of a virtual input/output server cluster
CN107147540A (zh) 高可用性系统中的故障处理方法和故障处理集群
CN109656742B (zh) 一种节点异常处理方法、装置及存储介质
CN108632057A (zh) 一种云计算服务器的故障恢复方法、装置及管理系统
CN105095001A (zh) 分布式环境下虚拟机异常恢复方法
CN111343219B (zh) 计算服务云平台
US11797287B1 (en) Automatically terminating deployment of containerized applications
US9195528B1 (en) Systems and methods for managing failover clusters
CN105051692A (zh) 通过隔离的自动化故障处理
CN114691304B (zh) 实现集群虚拟机高可用的方法和装置、设备和介质
CN118170324B (zh) 云硬盘迁移的方法及装置、电子设备和存储介质
CN111966599A (zh) 一种虚拟化平台可靠性测试方法、系统、终端及存储介质
CN111352662A (zh) 一种服务器启动顺序控制方法、系统、终端及存储介质
US12386649B2 (en) Server maintenance control device, server maintenance system, server maintenance control method, and program
CN115766405B (zh) 一种故障处理方法、装置、设备和存储介质
EP4254191A1 (en) Method and apparatus of implementing high availability of cluster virtual machine
CN109101253B (zh) 云计算系统中主机的管理方法和装置
CN114996055B (zh) 一种双机热备系统控制方法、装置、介质
CN110704160A (zh) 一种双机热备测试方法与装置
CN112527469B (zh) 一种云计算服务器的容错组合方法
US10365934B1 (en) Determining and reporting impaired conditions in a multi-tenant web services environment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23777733

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23777733

Country of ref document: EP

Kind code of ref document: A1