WO2021012785A1

WO2021012785A1 - Cloud disk mounting method and apparatus based on cloud storage, device, and storage medium

Info

Publication number: WO2021012785A1
Application number: PCT/CN2020/093106
Authority: WO
Inventors: 王鹏
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-07-22
Filing date: 2020-05-29
Publication date: 2021-01-28
Also published as: CN110515539A; CN110515539B

Abstract

The present application relates to the technical field of clouds, in particular to a cloud disk mounting method and apparatus based on cloud storage, a device and a storage medium. The method comprises: receiving a mounting request sent by a user, and determining whether a cloud disk read-write service is in an off-peak period; when the cloud disk read-write service is in the off-peak period, performing read-write lock control on a source cluster; and synchronously copying data of the source cluster to a target cluster. According to the present application, on the basis of a read-write lock between multi-service container cluster storage, the cluster requiring synchronous copying is locked to avoid an error in data when same is synchronously copied; and synchronous copying is carried out at the service off-peak period, such that cloud disk resources can be quickly used, and mounting deployment efficiency is improved.

Description

Cloud disk mounting method, device, equipment and storage medium based on cloud storage

This application requires the priority of a Chinese patent application filed with the Chinese Patent Office on July 22, 2019, with the application number 201910663198.4 and the invention title of "Cloud storage-based cloud disk mounting method, device, equipment and storage medium". The entire content is incorporated into this application by reference.

Technical field

This application relates to the field of cloud technology, and in particular to a cloud disk mounting method, device, device, and storage medium based on cloud storage.

Background technique

In recent years, cloud platforms have developed rapidly because they reduce user costs and improve resource utilization. The core of the cloud platform lies in the construction of virtualization technology. At present, the mainstream virtualization layer adopts a virtualization management platform (Kernel-based Virtual Machine, KVM for short). Cloud platform KVM usually uses clusters as the granularity, combined with a distributed storage system (ceph), to realize functions such as automatic mounting, fast online copying, and migration.

Currently, the cloud platform KVM on the market adopts a one-to-one relationship when mounting storage clusters to computing clusters, and one computing cluster corresponds to one storage cluster. The KVM host on the computing cluster is labeled host.pod1, and the ceph host on the storage cluster is labeled storage.pod1. A virtual image is established through the two labels to realize connection and storage. The inventor realized that this connection method leads to a tight coupling relationship between one computing cluster and one storage cluster. When there are two computing clusters and two storage clusters, this one-to-one relationship leads to any business container cluster and another The storage data of a business container cluster cannot be shared, causing troubles to the online and maintenance of the system.

technical problem

In view of this, it is necessary to provide a cloud disk mounting method, device, device, and storage medium based on cloud storage in response to the problem that data cannot be shared among multiple business container clusters and maintenance is difficult.

Technical solutions

A cloud disk mounting method based on cloud storage includes:

Receive the mount request sent by the user and determine whether the cloud disk read and write business is in a low peak period;

When the cloud disk read-write business is in a low peak period, perform read-write lock control on the source cluster;

Synchronously copy the data of the source cluster to the target cluster.

A cloud disk mounting device based on cloud storage includes:

The business judgment module is used to receive the mount request sent by the user and judge whether the cloud disk read and write business is in a low peak period;

The read-write lock module is used to perform read-write lock control on the source cluster when the cloud disk read-write business is in a low peak period;

The mounting module is used to synchronously copy the data of the source cluster to the target cluster.

A computer device includes a memory and a processor, and computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor executes the following steps:

Synchronously copy the data of the source cluster to the target cluster.

A storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the following steps:

Synchronously copy the data of the source cluster to the target cluster.

Beneficial effect

The above cloud storage-based cloud disk mounting method, device, equipment and storage medium include receiving a mounting request sent by a user and determining whether the cloud disk read and write business is in a low peak period; when the cloud disk read and write business is at a low peak During the period, the read-write lock control is performed on the source cluster; the data of the source cluster is synchronously copied to the target cluster. This application is based on the read-write lock between the storage of multi-service container clusters, which locks the clusters that need to be copied synchronously to avoid data errors during synchronous copying; the synchronous copying work is performed during low peak periods of business, so that cloud disk resources can be used quickly and improved Mount deployment efficiency.

Description of the drawings

FIG. 1 is a flowchart of a cloud disk mounting method based on cloud storage in an embodiment of the application;

FIG. 2 is a flowchart of step S1 in an embodiment of the application;

FIG. 3 is a flowchart of step S2 in an embodiment of this application;

FIG. 4 is a flowchart of step S3 in an embodiment of this application;

Fig. 5 is a structural diagram of a cloud disk mounting device based on cloud storage in an embodiment of the application.

The best mode of the invention

Figure 1 is a flowchart of a cloud disk mounting method based on cloud storage in an embodiment of this application. As shown in Figure 1, a cloud disk mounting method based on cloud storage includes the following steps:

Step S1, read and write business judgment: receive a mount request sent by a user, and judge whether the cloud disk read and write business is in a low peak period.

In this step, the mount request sent by the user can be obtained through the user hard disk configuration interface of the cloud portal website. The mount request includes the cloud hard disk information to be mounted and the target hard disk information. The user hard disk configuration interface can include the name, type, size, and usage status of the cloud hard disk that the user has allocated, as well as a mount button. Click the mount button to receive a mount request triggered by the user. After the cloud management system receives the mount request, it first determines whether the cloud disk read and write business is in a low peak period, and only in the low peak period can the subsequent automatic mounting step be performed.

In one embodiment, step S1, determining whether the cloud disk read and write business is in a low peak period, as shown in Figure 2, includes:

Step S101: Judging the current time: obtaining the current time, and judging whether the current time is in a preset low peak period.

After receiving the mount request sent by the user, first obtain the current time of the cloud management system, and determine whether the current time is in the preset low peak period based on the current time. The preset low peak period is the time when the cloud disk read and write business is less Segment, for example, between 12 AM and 3 AM. If the current time is not in this low peak period, it is determined that the cloud disk read and write business is in a non-low peak period, and subsequent automatic mounting steps are not performed. If the current time is in the preset low peak period, proceed to the next step.

Step S102: Calculate the index threshold: when the current time is in the low peak period, read the historical activity index of the cloud disk, calculate the average value of the activity index in the preset period, and multiply the average value of the activity index by the index weight to obtain the index threshold.

The activity indicators of cloud disks during peak business hours during the day and low business peaks at night differ greatly. For example, during the peak business hours during the day, the disk IOPS can reach 3000, while at night, the disk IOPS is only a few dozen. Therefore, the index threshold in this step is weighted after calculating the average value in the preset period.

The historical activity indicators in this step include the number of reads and writes per second and throughput of the disk, where the number of reads and writes per second of the disk is the disk IOPS. Disk IOPS and throughput are the main indicators to measure disk performance. For example, to read 10,000 files of 1KB in 10 seconds, IOPS=1000, throughput=1MB/S, and to read 1 file of 10MB in 0.2 seconds, IOPS=5, throughput=50MB/S. The preset period in this step may be the day before the current time, and the average value of the activity indicators is the average value of all historical activity indicators in the day before the current time. The preset period may also be the daytime business peak period corresponding to the current time, for example, between 6 o'clock and 18:00, the average activity index is the average value of the daytime business peak period. The indicator weight in this step is a preset percentage, such as 70%, and the indicator threshold is 70% of the average value of the activity indicator.

Before this step, it also includes: through a preset monitoring script, according to the sampling period, read the number of reads and writes per second of the cloud disk, the throughput of the cloud disk, and compare the sampling time, the number of reads and writes per second, and the throughput. The historical activity indicators stored as cloud disks are associated together.

Disk IOPS and throughput indicators can be used to test disk IOPS and throughput in different situations through preset monitoring scripts such as test benchmark tools, mainly Iometer, IoZone, FIO, etc. In this step, the load characteristics of the data can be determined according to different application systems, and reasonable IOPS indicators can be selected for measurement. The sampling period can be preset, such as a time period of 1 minute, 5 minutes, or 10 minutes.

Step S103: Determine the low peak period: read the activity index of the cloud disk at the current time and determine whether the activity index at the current time is lower than the index threshold. If it is lower, it is judged that the cloud disk read and write business is in the low peak period.

The activity indicators at the current time are obtained using the same monitoring script as in step S102, that is, the disk IOPS and disk throughput at the current time are obtained. Compare the disk IOPS at the current time with the index threshold corresponding to the disk IOPS calculated in step S102, compare the disk throughput at the current time with the index threshold corresponding to the disk throughput calculated in step S102, and only the disk IOPS at the current time When both the disk throughput and the disk throughput are lower than the corresponding indicator threshold, the cloud disk read and write business is determined to be in a low peak period.

This embodiment is a step-by-step method for judging whether the current time is in the low peak period and whether the activity index corresponding to the current time is lower than the index threshold. It is determined whether the disk read and write business is in the low peak period. Only under the premise of the low peak period, Proceed to the subsequent automatic mounting steps.

Step S2, read-write lock control: when the cloud disk read-write business is in a low peak period, read-write lock control is performed on the source cluster.

This step is only performed when the cloud disk read and write business is in a low peak period. Before automatic mounting, the read-write lock control is also performed on the source cluster.

In one embodiment, step S2, as shown in FIG. 3, includes:

Step S201: Define the source cluster: when the cloud disk read and write business is in a low peak period, one of the cloud disk or the target disk in the mount request is defined as the source cluster, and the other is defined as the target cluster.

In the mount request sent by the user, the cloud hard disk requested by the user and the corresponding target hard disk are regarded as two clusters and form a group. The two clusters in the same group can be either the source cluster or the target cluster. Two clusters perform differential copies. In this step, one of them can be randomly determined as the source cluster and the other as the target cluster.

Step S202, locking the source cluster: adopting the preset service mode of the metadata server of the distributed file storage system to control the source cluster with three types of locks: read, write, and mutual exclusion.

In the read-write lock control, the service mode of the metadata server (ceph mds) of the distributed file storage system is used to implement three types of locks: read (rdlock), write (wrlock), and mutually exclusive (xlock).

During the read-write lock control period, if the user read data is received, the data stored in the business container (pod) in the source cluster is read, and the read data is normal at this time. If the user's write data is received, the data will be written to the cache without placing it in the disk. After the automatic mounting step is completed, the write data will be placed in the disk.

In this embodiment, before the data of the source cluster is quickly mounted, the source cluster is first controlled by the read-write lock to prevent the user from performing modification operations.

Step S3, automatic mounting: synchronously copy the data of the source cluster to the target cluster.

This step realizes the purpose of automatic mounting by automatically migrating and copying the difference data of the source cluster to the target cluster.

In one embodiment, step S3, as shown in FIG. 4, includes:

Step S301, data query and comparison: read the data in the source cluster through the SQL structured query language, compare it with the target cluster, and if the target cluster does not contain data, store the data in the cache.

When performing this step, the ceph monitor of the distributed file storage system is used to perform data object attributes and maintain metadata. So that only one thread can obtain the monitor of the data object at the same time, after a thread owns the monitor of the object, it can call notify() and notifyall() of the object to wake up and execute the thread. Ceph monitor can maintain the consistency of the cluster and avoid unnecessary congestion and data disorder.

Step S302, data copy: when all the data with differences are stored in the cache, all the data in the cache is written into the target cluster.

Since the target cluster does not perform read-write lock control, this step is to copy all the difference data to the target cluster at one time after storing all the difference data in the cache, effectively avoiding data confusion. After completing the synchronous copy of the source cluster data to the target cluster, define the target cluster as the source cluster, define the source cluster as the target cluster, and perform the read-write lock control on the source cluster again and the subsequent synchronous copy of the source cluster data to the target Steps in the cluster.

This embodiment uses SQL to read and compare the difference data in the two clusters, and puts all the difference data into the cache. Only after the difference data is completely stored, all the data in the cache is written to the target cluster. The SQL language and Cache technology realizes the purpose of synchronously copying the data of the source cluster to the target cluster.

In an embodiment, after step S3, it further includes:

Receive the disk creation request sent by the user, obtain the creation parameters in the disk creation request, and call the partition creation interface to create a new disk with the creation parameters.

In this step, when receiving a disk creation request, the user's request is obtained through the user hard disk configuration interface of the cloud portal website. Creation parameters include disk name, disk type, disk capacity, IOPS parameters, number of disks, etc. For example, after receiving a disk creation request with a creation capacity of 60GB and 1120IOPS parameters sent by the user, a new disk is created by calling the create partition interface (createVolume interface). When creating a new disk, you can also obtain the user name of the user as the creator, and display it to the user for viewing along with the parameters corresponding to the created disk.

Receive the disk adjustment request sent by the user, obtain the adjustment parameters in the adjustment disk request, and call the partition size adjustment interface to adjust the parameters of the disk.

In this step, when receiving a disk adjustment request, the user's request is obtained through the user hard disk configuration interface of the cloud portal website. The user hard disk configuration interface displays the created cloud hard drives, including the disk name, disk type, disk size, status, creator, and operable content of each cloud hard disk. After receiving the disk adjustment request sent by the user, obtain the adjustment parameters input by the user through the modification interface. The adjustment parameters include the cloud hard disk name or instance, and the adjusted capacity. By calling the resizeVolume interface, adjust the cloud hard disk name or the disk capacity of the instance. It is the adjusted capacity. For example, the user can implement disk adjustment requests by modifying the interface, such as requesting to modify the disk capacity of the cloud disk named SC-T0600256 to 1000GB. The user clicks the confirmation button in the modification interface to trigger the disk adjustment request. The cloud management system receives the above disk adjustment request, obtains the adjustment parameters, and adjusts the SC-T0600256 disk capacity to 1000GB by calling the resize partition interface (resizeVolume interface).

In this embodiment, after the automatic mounting is completed, cloud disk creation and capacity adjustment are performed through the createVolume interface and the resizeVolume interface, respectively, to provide users with more complete cloud disk performance adjustment requirements, and there is no need to stop during the creation and capacity adjustment. , To ensure business continuity.

In this embodiment, the cloud disk mounting method based on cloud storage is automatically mounted only when the cloud disk read and write business is in a low peak period, which ensures the reasonable use of cloud disk resources and improves the mounting efficiency; Before mounting, read-write lock control is also performed on the source cluster to prevent users from modifying data and avoid data confusion; during automatic mounting, the difference data is written to the target cluster at one time through SQL language and caching technology. Not only achieves the purpose of fast mounting, but the system does not need to be shut down, ensuring business continuity.

In one embodiment, a cloud disk mounting device based on cloud storage is proposed, as shown in FIG. 5, including the following modules:

The read-write lock module is used to control the read-write lock of the source cluster when the cloud disk read-write business is in a low peak period;

The mount module is used to synchronously copy the data of the source cluster to the target cluster.

In one embodiment, a computer device is proposed, including a memory and a processor, and computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor executes the computer-readable instructions to implement the above-mentioned The steps in the cloud disk mounting method of cloud storage include:

Synchronously copy the data of the source cluster to the target cluster.

In one embodiment, a storage medium storing computer-readable instructions is provided. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the above-mentioned cloud storage-based cloud disk attachment. The steps in the loading method include:

Synchronously copy the data of the source cluster to the target cluster.

Wherein, the storage medium may be a non-volatile storage medium. It can also be a volatile storage medium.

A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the storage medium can include: Read only memory (ROM , Read Only Memory), random access memory (RAM, Random Access Memory), magnetic or optical disks, etc.

The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, All should be considered as the scope of this specification.

Claims

A cloud disk mounting method based on cloud storage, which includes:

Receive the mount request sent by the user and determine whether the cloud disk read and write business is in a low peak period;

When the cloud disk read-write business is in a low peak period, perform read-write lock control on the source cluster;

Synchronously copy the data of the source cluster to the target cluster.
The cloud disk mounting method based on cloud storage according to claim 1, wherein said determining whether the cloud disk read and write business is in a low peak period comprises:

Get the current time and determine whether the current time is in the preset low peak period;

When the current time is in the low peak period, read the historical activity indicators of the cloud disk, calculate the average value of the activity indicators in a preset period, and multiply the average value of the activity indicators by the indicator weight to obtain the indicator threshold;

Read the activity index of the cloud disk at the current time, and determine whether the activity index at the current time is lower than the index threshold, and if it is lower, it is determined that the cloud disk read-write business is in a low peak period.
The method for mounting cloud disks based on cloud storage according to claim 2, wherein said reading historical activity indicators of said cloud disks, calculating an average value of activity indicators in a preset period, and calculating the average value of said activity indicators Before multiplying the index weight to obtain the index threshold, the method further includes:

Through a preset monitoring script, read the number of reads and writes per second of the cloud disk and the throughput of the cloud disk respectively according to the sampling period, and associate the sampling time, the number of reads and writes per second and the throughput together Stored as a historical activity indicator of the cloud disk.
The method for mounting cloud disks based on cloud storage according to claim 1, wherein said performing read-write lock control on the source cluster when the cloud disk read-write business is in a low peak period comprises:

When the cloud disk read and write business is in a low peak period, define one of the cloud hard disk or the target hard disk in the mount request as the source cluster, and the other as the target cluster;

Using the preset service mode of the metadata server of the distributed file storage system, the source cluster is controlled by three types of locks: read, write, and mutual exclusion.
The method for mounting cloud disks based on cloud storage according to claim 1, wherein said synchronously copying the data of the source cluster to the target cluster comprises:

Read the data in the source cluster through the SQL structured query language, compare it with the target cluster, and if the target cluster does not contain the data, store the data in a cache;

When all the data with differences are stored in the cache, all the data in the cache is written into the target cluster.
The method for mounting cloud disks based on cloud storage according to claim 1, wherein after the synchronous copying of the data of the source cluster to the target cluster, the method further comprises:

Receive the disk creation request sent by the user, obtain the creation parameters in the disk creation request, call the partition creation interface, and create a new disk with the creation parameters.
The method for mounting cloud disks based on cloud storage according to claim 1, wherein after synchronously copying the data of the source cluster to the target cluster, the method further comprises:

Receive the disk adjustment request sent by the user, obtain the adjustment parameters in the adjustment disk request, call the partition size adjustment interface, and perform parameter adjustment on the disk using the adjustment parameters.
A cloud disk mounting device based on cloud storage, which includes:

The business judgment module is used to receive the mount request sent by the user and judge whether the cloud disk read and write business is in a low peak period;

The read-write lock module is used to perform read-write lock control on the source cluster when the cloud disk read-write business is in a low peak period;

The mounting module is used to synchronously copy the data of the source cluster to the target cluster.
A computer device, which includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes a cloud storage-based Cloud disk mounting methods include:

Receive the mount request sent by the user and determine whether the cloud disk read and write business is in a low peak period;

When the cloud disk read-write business is in a low peak period, perform read-write lock control on the source cluster;

Synchronously copy the data of the source cluster to the target cluster.
The computer device according to claim 9, wherein said determining whether the cloud disk read and write service is in a low peak period comprises:

Get the current time and determine whether the current time is in the preset low peak period;

When the current time is in the low peak period, read the historical activity index of the cloud disk, calculate the average value of the activity index in a preset period, and multiply the average value of the activity index by the index weight to obtain an index threshold;

Read the activity index of the cloud disk at the current time, and determine whether the activity index at the current time is lower than the index threshold, and if it is lower, it is determined that the cloud disk read-write business is in a low peak period.
The computer device according to claim 10, wherein the reading the historical activity index of the cloud disk, calculating the average value of the activity index in a preset period, and multiplying the average value of the activity index by the index weight to obtain the index Before the threshold, the method further includes:

Through a preset monitoring script, read the number of reads and writes per second of the cloud disk and the throughput of the cloud disk respectively according to the sampling period, and associate the sampling time, the number of reads and writes per second and the throughput together Stored as a historical activity indicator of the cloud disk.
9. The computer device according to claim 9, wherein said performing read-write lock control on the source cluster when the cloud disk read-write service is in a low peak period comprises:

When the cloud disk read-write business is in a low peak period, define one of the cloud hard disk or the target hard disk in the mount request as the source cluster, and the other as the target cluster;

Using the preset service mode of the metadata server of the distributed file storage system, the source cluster is controlled by three types of locks: read, write, and mutual exclusion.
The computer device according to claim 9, wherein the synchronously copying the data of the source cluster to the target cluster comprises:

Read the data in the source cluster through the SQL structured query language, compare it with the target cluster, and if the target cluster does not contain the data, store the data in a cache;

When all the data with differences are stored in the cache, all the data in the cache is written into the target cluster.
The computer device according to claim 9, wherein, after the synchronously copying the data of the source cluster to the target cluster, the method further comprises:

Receive the disk creation request sent by the user, obtain the creation parameters in the disk creation request, call the partition creation interface, and create a new disk with the creation parameters.
9. The computer device according to claim 9, wherein, after synchronously copying the data of the source cluster to the target cluster, the method further comprises:

Receive the disk adjustment request sent by the user, obtain the adjustment parameters in the adjustment disk request, call the partition size adjustment interface, and perform parameter adjustment on the disk using the adjustment parameters.
A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, one or more processors execute a cloud disk mounting method based on cloud storage, include:

Receive the mount request sent by the user and determine whether the cloud disk read and write business is in a low peak period;

When the cloud disk read-write business is in a low peak period, perform read-write lock control on the source cluster;

Synchronously copy the data of the source cluster to the target cluster.
The storage medium storing computer readable instructions according to claim 16, wherein said determining whether the cloud disk read and write business is in a low peak period comprises:

Get the current time and determine whether the current time is in the preset low peak period;

When the current time is in the low peak period, read the historical activity indicators of the cloud disk, calculate the average value of the activity indicators in a preset period, and multiply the average value of the activity indicators by the indicator weight to obtain the indicator threshold;

Read the activity index of the cloud disk at the current time, and determine whether the activity index at the current time is lower than the index threshold, and if it is lower, it is determined that the cloud disk read-write business is in a low peak period.
The storage medium storing computer-readable instructions according to claim 17, wherein said reading the historical activity index of the cloud disk, calculating the average value of the activity index in a preset period, and calculating the average value of the activity index Before multiplying the index weight to obtain the index threshold, the method further includes:

Through a preset monitoring script, read the number of reads and writes per second of the cloud disk and the throughput of the cloud disk respectively according to the sampling period, and associate the sampling time, the number of reads and writes per second and the throughput together Stored as a historical activity indicator of the cloud disk.
The storage medium storing computer-readable instructions according to claim 16, wherein said performing read-write lock control on the source cluster when the cloud disk read-write service is in a low peak period comprises:

When the cloud disk read and write business is in a low peak period, define one of the cloud hard disk or the target hard disk in the mount request as the source cluster, and the other as the target cluster;

Using the preset service mode of the metadata server of the distributed file storage system, the source cluster is controlled by three types of locks: read, write, and mutual exclusion.
The storage medium storing computer-readable instructions according to claim 16, wherein said synchronously copying the data of the source cluster to the target cluster comprises:

Read the data in the source cluster through the SQL structured query language, compare it with the target cluster, and if the target cluster does not contain the data, store the data in a cache;

When all the data with differences are stored in the cache, all the data in the cache is written into the target cluster.