CN117891612A - Snapshot cloning method, device and storage medium - Google Patents

Abstract

The application discloses a snapshot cloning method, a snapshot cloning device and a storage medium, relates to the technical field of communication, and is used for solving the problem that snapshot cloning speed is low when a plurality of requests are processed. The method comprises the following steps: determining a cloning task queue; the cloning task queue comprises at least one cloning task; determining one or more target cloning tasks meeting preset conditions in a cloning task queue; matching clone threads for each target clone task from a clone thread pool; each target clone task is processed based on the matched clone threads.

Description

Snapshot cloning method, device and storage medium

Technical Field

The present disclosure relates to the field of communications technologies, and in particular, to a snapshot cloning method, a snapshot cloning device, and a storage medium.

Background

Cephfs provides directory snapshot capability, can snapshot directory tree, and can restore when a certain file is damaged. Many users wish to clone a directory that is identical in data but independent from one file system to another through a directory snapshot.

Currently, in the general technology, snapshot cloning is realized by means of a third party platform, wherein the third party platform converts a user request, and then the snapshot cloning is completed through a cp command. However, cp commands are typically used to copy one or more source files into a specified file or directory, so that a third party platform can only process one clone request at a time, and after this clone is completed, other snapclone can be performed, resulting in a slower speed of snapclone when processing multiple requests.

Disclosure of Invention

The application provides a snapshot cloning method, a snapshot cloning device and a storage medium, which are used for solving the problem that snapshot cloning speed is low when a plurality of requests are processed in a general technology.

In order to achieve the above purpose, the present application adopts the following technical scheme:

in a first aspect, a snapshot cloning method is provided, including: determining a cloning task queue; the cloning task queue comprises at least one cloning task; determining one or more target cloning tasks meeting preset conditions in a cloning task queue; matching clone threads for each target clone task from a clone thread pool; the clone thread pool comprises at least one clone thread for processing clone tasks; each target clone task is processed based on the matched clone threads.

Optionally, the method for processing each target cloning task based on the matched cloning threads comprises the following steps: determining a cloning stage of a target cloning task based on the matched cloning threads; each target cloning task is processed based on its cloning stage.

Optionally, the method for processing each target cloning task based on the cloning stage comprises the following steps: when the cloning stage of the target cloning task is a cloning preparation stage, cloning a source directory of the target cloning task to a target directory; when the cloning stage of the target cloning task is a cloning progress stage, cloning the directory entry into the target directory based on the type of the directory entry in the source directory; when the cloning stage of the target cloning task is a cloning completion stage, the target cloning task is removed from the cloning task queue.

Optionally, the method for cloning the source catalog of the target cloning task to the target catalog comprises the following steps: acquiring a source directory of a target cloning task and directory attributes of the source directory; creating a target directory in the target file system based on the target path name, and synchronizing directory attributes of the source directory to the target directory; the target path name is used to determine the path of the target directory.

Optionally, when the type of the directory entry is a subdirectory, the method for cloning the directory entry into the target directory based on the type of the directory entry in the source directory includes: obtaining subdirectories in a source directory and directory attributes of the subdirectories; creating a target subdirectory under the target directory, and synchronizing the directory attributes of the subdirectory to the target subdirectory.

Optionally, when the type of the directory entry is a file, the method for cloning the directory entry into the target directory based on the type of the directory entry in the source directory includes: acquiring files in a source directory, file data of the files and file attributes; creating a target file under the target directory, cloning file data into the target file, and synchronizing file attributes of the file to the target file.

Optionally, the method for determining the cloning task queue comprises the following steps: obtaining snapshot cloning request information; based on the snapshot cloning request information, a cloning task is determined and added to a cloning task queue.

Optionally, before determining the cloning task queue, the method further includes: receiving snapshot cloning request information, generating an insertion statement based on the snapshot cloning request information, and inserting the insertion statement into a cloning table of a first database; the clone table includes: snapshot cloning request information; generating an insertion script based on the clone table, and writing the insertion script into a second database; the second database is used for storing snapshot cloning request information; obtaining snapshot clone request information includes: and obtaining snapshot cloning request information from the second database.

Optionally, the snapshot cloning method further includes: acquiring the utilization rate of each disk in the plurality of disks, and determining the disk with the utilization rate greater than a preset threshold value in the plurality of disks as a target disk; determining the proportion of the number of the target disks to the number of the plurality of disks; adjusting a cloning time interval of each cloning thread in the plurality of disks based on the proportion; the cloning time interval is used to determine the clone thread that matches for each target clone task.

In a second aspect, there is provided a snapshot cloning device comprising: a processing unit; a processing unit for determining a clone task queue; the cloning task queue comprises at least one cloning task; the processing unit is also used for determining one or more target cloning tasks meeting preset conditions in the cloning task queue; the processing unit is also used for matching the clone thread for each target clone task from the clone thread pool; the clone thread pool comprises at least one clone thread for processing clone tasks; and the processing unit is also used for processing each target cloning task based on the matched cloning threads.

Optionally, the processing unit is specifically configured to: determining a cloning stage of a target cloning task based on the matched cloning threads; each target cloning task is processed based on its cloning stage.

Optionally, the processing unit is specifically configured to: when the cloning stage of the target cloning task is a cloning preparation stage, cloning a source directory of the target cloning task to a target directory; when the cloning stage of the target cloning task is a cloning progress stage, cloning the directory entry into the target directory based on the type of the directory entry in the source directory; when the cloning stage of the target cloning task is a cloning completion stage, the target cloning task is removed from the cloning task queue.

Optionally, the snapshot cloning device further comprises: a communication unit; the communication unit is specifically used for: acquiring a source directory of a target cloning task and directory attributes of the source directory; the processing unit is specifically used for: creating a target directory in the target file system based on the target path name, and synchronizing directory attributes of the source directory to the target directory; the target path name is used to determine the path of the target directory.

Optionally, when the type of the directory entry is a subdirectory, the communication unit is specifically configured to: obtaining subdirectories in a source directory and directory attributes of the subdirectories; the processing unit is specifically used for: creating a target subdirectory under the target directory, and synchronizing the directory attributes of the subdirectory to the target subdirectory.

Optionally, when the type of the directory entry is a file, the communication unit is specifically configured to: acquiring files in a source directory, file data of the files and file attributes; the processing unit is specifically used for: creating a target file under the target directory, cloning file data into the target file, and synchronizing file attributes of the file to the target file.

Optionally, the communication unit is specifically configured to: obtaining snapshot cloning request information; the processing unit is specifically used for: based on the snapshot cloning request information, a cloning task is determined and added to a cloning task queue.

Optionally, before determining the clone task queue, the communication unit is further configured to: receiving snapshot cloning request information; the processing unit is further used for: generating an insertion statement based on the snapshot cloning request information, and inserting the insertion statement into a cloning table of the first database; the clone table includes: snapshot cloning request information; the processing unit is further used for: generating an insertion script based on the clone table, and writing the insertion script into a second database; the second database is used for storing snapshot cloning request information; wherein obtaining snapshot clone request information includes: and obtaining snapshot cloning request information from the second database.

Optionally, the snapshot cloning device, the communication unit is further configured to: acquiring the utilization rate of each disk in a plurality of disks; the processing unit is further used for: determining a disk with the utilization rate larger than a preset threshold value in the plurality of disks as a target disk; the processing unit is further used for: determining the proportion of the number of the target disks to the number of the plurality of disks; the processing unit is further used for: adjusting a cloning time interval of each cloning thread in the plurality of disks based on the proportion; the cloning time interval is used to determine the clone thread that matches for each target clone task.

In a third aspect, a snapshot cloning device is provided, comprising a memory and a processor; the memory is used for storing computer execution instructions, and the processor is connected with the memory through a bus; when the snapshot cloning device is running, the processor executes computer-executable instructions stored in the memory to cause the snapshot cloning device to perform the snapshot cloning method described in the first aspect.

The snapshot cloning device may be a network device or may be a part of a device in a network device, such as a chip system in a network device. The system-on-chip is configured to support the network device to implement the functions involved in the first aspect and any one of its possible implementations, for example, to obtain, determine, and send data and/or information involved in the above-mentioned snapshot cloning method. The chip system includes a chip, and may also include other discrete devices or circuit structures.

In a fourth aspect, there is provided a computer readable storage medium comprising computer executable instructions which, when run on a computer, cause the computer to perform the snapshot cloning method of the first aspect.

In a fifth aspect, there is also provided a computer program product comprising computer instructions which, when run on a snapshot cloning device, cause the snapshot cloning device to perform the snapshot cloning method as described in the first aspect above.

It should be noted that the above-mentioned computer instructions may be stored in whole or in part on the first computer readable storage medium. The first computer readable storage medium may be packaged together with the processor of the snapclone device, or may be packaged separately from the processor of the snapclone device, which is not limited in this application.

The description of the second, third, fourth and fifth aspects of the present application may refer to the detailed description of the first aspect; the advantages of the second aspect, the third aspect, the fourth aspect and the fifth aspect may be referred to as analysis of the advantages of the first aspect, and will not be described here.

In this application, the names of the above-mentioned snapclone devices do not constitute limitations on the devices or functional modules themselves, which may appear under other names in an actual implementation. Insofar as the function of each device or function module is similar to the present application, it is within the scope of the claims of the present application and the equivalents thereof.

These and other aspects of the present application will be more readily apparent from the following description.

The technical scheme provided by the application at least brings the following beneficial effects:

based on any one of the above aspects, the present application provides a snapshot cloning method, which can determine one or more target cloning tasks meeting a preset condition in a cloning task queue, and match a cloning thread for each target cloning task from a cloning thread pool. And finally, processing each target cloning task based on the matched cloning threads. Compared with the general technology, the method and the device have the advantages that the clone threads can be matched for a plurality of target clone tasks at the same time, and then the plurality of target clone tasks are processed at the same time, so that the processing efficiency can be improved, the effective utilization of system resources is ensured, and the speed of processing a plurality of snapshot clone tasks is improved.

Drawings

FIG. 1 is a schematic diagram of a snapclone system according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a snapclone device according to an embodiment of the present application;

fig. 3 is a schematic diagram of a communication device according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating a snapshot cloning method according to an embodiment of the present application;

FIG. 5 is a second flowchart of a snapshot cloning method according to an embodiment of the present application;

FIG. 6 is a flowchart of a snapshot cloning method according to an embodiment of the present application;

FIG. 7 is a flowchart of a snapshot cloning method according to an embodiment of the present application;

FIG. 8 is a flowchart fifth embodiment of a snapshot cloning method according to the present disclosure;

FIG. 9 is a flowchart of a snapshot cloning method according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a snapclone device according to an embodiment of the present application.

Detailed Description

The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", and the like are used to distinguish the same item or similar items having substantially the same function and effect, and those skilled in the art will understand that the terms "first", "second", and the like are not limited in number and execution order.

The following explains the terms related to the embodiments of the present application, so as to facilitate the understanding of the reader.

(1) Ceph: ceph is a unified, distributed file system designed for excellent performance, reliability and scalability, and can provide object, file, and block storage at the same time.

(2) Cephfs (Ceph file system): cephfs is a distributed file system provided by Ceph.

(3) Ceph-fuse: ceph-fuse is a client of the distributed file system.

(4) MDS: the MDS can be used for managing metadata of a file system, can control interaction between a client and an OSD object, and provides metadata for the client, including organization relation of files and catalogs, OSD corresponding to each file and the like.

(5) OSD: ceph's storage node service with data storage, intelligent distribution and object data management functions

(6) Reliable, automated, distributed object storage (reliable autonomous distributed object storage, rados): rados is a storage base of Ceph, and data of Ceph are stored in Rados clusters composed of OSD.

(7) monitor: the monitoring service of the Ceph distributed file system is used for storing the topological graph of the cluster and perceiving the distribution of storage service nodes.

(8) Manager (MGR): the MGR component is a component in the Ceph distributed storage system responsible for coordinating, maintaining, and managing the state and metadata of the entire cluster. The manager is responsible for handling the client's request and mapping the client's request to the correct object storage location. By looking up the metadata table, the manager can determine the OSD node where the object is located and forward the request to the corresponding OSD node.

(9) util index: the percentage of time that a disk processes I/O represents the disk Input/Output (I/O) utilization.

The foregoing is a simplified description of some of the concepts involved in the embodiments of this application.

Ceph can provide directory snapshot function to users, allowing directory subtrees to be snapshot, so that when a certain file is damaged, users can restore the file through snapshot. Many users now also wish to clone an identical but independent directory through a directory snapshot between different file systems.

At present, the snapshot cloning in the general technology needs to be realized by means of a third party platform (such as an application cloud platform), the application cloud platform can receive a snapshot cloning request of a user, and then perform some logic processing on the request, such as recording a database, setting a task state and the like, and then send a cp command to Cephfs. Cephfs obtains the cp command and copies the snapshot 1 in the file system A to the directory corresponding to the file system B.

However, cp commands are typically used to copy one or more source files into a specified file or directory, so that a third party platform can only process one clone request at a time, and after this clone is completed, other snapclone can be performed, resulting in a slower speed of snapclone when processing multiple requests.

The embodiment of the application provides a snapshot cloning method, which can determine one or more target cloning tasks meeting preset conditions in a cloning task queue and match a cloning thread for each target cloning task from a cloning thread pool. And finally, processing each target cloning task based on the matched cloning threads. Compared with the general technology, the method and the device have the advantages that the clone threads can be matched for a plurality of target clone tasks at the same time, and then the plurality of target clone tasks are processed at the same time, so that the processing efficiency can be improved, the effective utilization of system resources is ensured, and the speed of processing a plurality of snapshot clone tasks is improved.

The snapshot cloning method is suitable for a snapshot cloning system. FIG. 1 shows a schematic diagram of a snapshot cloning system. As shown in fig. 1, the snapshot cloning system 100 includes: snapshot cloning device 101, user terminal 102.

In one embodiment, the user terminal 102 sends the snapclone request information to the snapclone device 101, and the snapclone device 101 receives the snapclone request information. The snapshot cloning device 101 determines a cloning task based on the snapshot cloning request information, and adds the cloning task to a cloning task queue. The snapshot cloning device 101 again determines target cloning tasks in the cloning task queue that satisfy the preset condition, and matches the cloning threads for each target cloning task from the cloning thread pool. Finally, the snapshot cloning device processes each target cloning task through the matched cloning threads.

Alternatively, the snapclone device 101 and the user terminal 102 may be connected by a wired or wireless means.

Alternatively, one snapclone device 101 may be connected to a plurality of user terminals 102, and one user terminal 102 may also be connected to a plurality of snapclone devices 101. For ease of understanding, fig. 1 illustrates an example in which a snapshot cloning device 101 is connected to a user terminal 102.

Alternatively, the user terminal 102 may be a handheld device with wireless connectivity, or a wireless terminal connected to other processing devices of a wireless modem, or may be a wired terminal. For example, smart devices such as cell phones, personal computers (personal computer, PCs), desktop computers, tablet computers, notebook computers, netbooks, personal digital assistants (personal digital assistant, PDAs), and the like, which are not limited in this embodiment.

As shown in fig. 2, the snapclone device 101 may include: MGR110 and a plurality of file systems. Fig. 2 illustrates an example in which the snapclone device 101 includes two file systems (file system 111, file system 112). The MGR110 includes a user command interface module 113, a clone sub-tree module 114, and a load sensing module 115.

The MGR110 is configured to receive the snapshot cloning request information from the user terminal 102, and clone the source directory in the file system 111 into the file system 112 based on the snapshot cloning request information, so as to obtain the target directory.

Optionally, the user command interface module 113 is configured to receive snapshot cloning request information from the user terminal 102.

Optionally, the clone sub-tree module 114 is configured to obtain the source directory from the file system 111 through the clone thread pool based on the snapshot clone request information. The source directory is then cloned into the file system 112 to obtain the target directory.

Optionally, the load sensing module 115 is configured to obtain the usage rate of each disk in the snapclone device 101, and adjust the cloning time interval of each thread in the disk based on the usage rate.

In particular implementation, the snapclone device 101 and the user terminal 102 in fig. 1 may each adopt the constituent structure shown in fig. 3, or include the components shown in fig. 3. Fig. 3 is a schematic diagram of a communication device provided in an embodiment of the present application, where the communication device may be a snapclone device 101 or a chip or a system on chip in the snapclone device 101. Alternatively, the communication device may be the user terminal 102 or a chip or a system on a chip in the user terminal 102. As shown in fig. 3, the communication device may include a processor 301 and a communication line 302.

Further, the communication device may further comprise a communication interface 303 and a memory 304. The processor 301, the memory 304, and the communication interface 303 may be connected by a communication line 302.

The processor 301 is a CPU, general-purpose processor, network processor (network processor, NP), digital signal processor (digital signal processing, DSP), microprocessor, microcontroller, programmable logic device (programmable logic device, PLD), or any combination thereof. The processor 301 may also be any other device having processing functions, such as, without limitation, a circuit, a device, or a software module.

A communication line 302 for communicating information between the various components included in the communication device.

A communication interface 303 for communicating with other devices or other communication networks. The other communication network may be an ethernet, a radio access network (radio access network, RAN), a wireless local area network (wireless local area networks, WLAN), etc. The communication interface 303 may be a module, a circuit, a communication interface, or any device capable of enabling communication.

Memory 304 for storing instructions. Wherein the instructions may be computer programs.

The memory 304 may be, but not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and/or instructions, a random access memory (random access memory, RAM) or other type of dynamic storage device capable of storing information and/or instructions, an EEPROM, a CD-ROM (compact disc read-only memory) or other optical disk storage, an optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, etc.

It should be noted that the memory 304 may exist separately from the processor 301 or may be integrated with the processor 301. Memory 304 may be used to store instructions or program code or some data, etc. Memory 304 may be located within the communication device or external to communication device 300, without limitation. The processor 301 is configured to execute instructions stored in the memory 304 to implement a snapshot cloning method provided in the following embodiments of the present application.

In one example, processor 301 may include one or more CPUs, e.g., CPU0 and CPU1.

As an alternative implementation, the communication device includes a plurality of processors.

As an alternative implementation, the communication apparatus further comprises an output device and an input device. The output device is illustratively a display screen, speaker (spaker) or the like, and the input device is a keyboard, mouse, microphone or joystick or the like.

It should be noted that the constituent structures shown in fig. 3 do not constitute limitations of the respective apparatuses in fig. 1 and 3, and that the respective apparatuses in fig. 1 and 3 may include more or less components than illustrated, or may combine some components, or may be arranged differently, in addition to the components shown in fig. 3.

In the embodiment of the application, the chip system may be formed by a chip, and may also include a chip and other discrete devices.

Further, actions, terms, etc. referred to between embodiments of the present application may be referred to each other without limitation. In the embodiment of the present application, the name of the message or the name of the parameter in the message, etc. interacted between the devices are only an example, and other names may also be adopted in the specific implementation, and are not limited.

The snapshot cloning method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings. As shown in fig. 4, the snapshot cloning method includes:

s401, the snapshot cloning device determines a cloning task queue.

Wherein the clone task queue comprises at least one clone task.

In one possible implementation, a snapclone device obtains snapclone request information from a database, packages each snapclone request information into one clone task, and adds the clone task to a clone task queue. The cloning task queue is used for storing cloning tasks and waiting for the snapshot cloning device to process the cloning tasks.

Alternatively, the cloning task may be a cloning job task.

S402, the snapshot cloning device determines one or more target cloning tasks meeting preset conditions in a cloning task queue.

Optionally, the preset conditions include: the current time satisfies a preset cloning time of the cloning task in the cloning task queue.

In one possible implementation manner, the snapshot cloning device detects whether a cloning task exists in the cloning task queue, wherein the current time of the cloning task meets the preset cloning time of the cloning task in the cloning task queue, and if so, the snapshot cloning device determines the cloning task meeting the preset condition as a target cloning task.

Illustratively, the cloning task queue includes: a first cloning task, a second cloning task, and a third cloning task. The preset cloning time of the first cloning task is ten points in the evening to six points in the morning, the preset cloning time of the second cloning task is four points in the morning to seven points in the morning, and the preset cloning time of the third cloning task is six points in the morning to seven points in the morning. The snapshot cloning device determines that the current time is five points in the morning, and at the moment, the preset cloning time of the first cloning task and the second cloning task is met, and the further snapshot cloning device can determine that the first cloning task and the second cloning task are target cloning tasks.

S403, the snapshot cloning device matches the cloning thread for each target cloning task from the cloning thread pool.

Wherein the clone thread pool comprises at least one clone thread for processing clone tasks.

In one possible implementation, after the snapclone device determines the target clone device in the clone task queue, the snapclone device matches clone threads for each target clone task from the clone thread pool, and the matched clone threads acquire corresponding target clone tasks from the clone task queue for processing.

S404, the snapshot cloning device processes each target cloning task based on the matched cloning threads.

In one possible implementation, the snapshot cloning device obtains the target cloning task from the cloning task queue based on the cloning thread that each target cloning task matches; the snapshot cloning device runs the target cloning task through the corresponding cloning thread and adds the target cloning task to a cloning running queue, wherein the cloning running queue is used for recording the target cloning task which is being cloned.

Optionally, elements in the clone operation queue are stored in units of triples consisting of a source file system, a target clone task and a thread identifier, and each triplet represents a corresponding target clone task.

Illustratively, the snapshot cloning device matches a first cloning thread for a first cloning task and a second cloning thread for a second cloning task. The first clone thread obtains a first clone task from the clone task queue, starts running the first clone task and adds the first clone task to the clone run queue. The second clone thread obtains a second clone task from the clone task queue, starts running the second clone task and adds the second clone task to the clone run queue.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as shown in S401 to S404, the present application may determine one or more target clone tasks in the clone task queue that satisfy the preset condition, and match the clone thread for each target clone task from the clone thread pool. And finally, processing each target cloning task based on the matched cloning threads. Compared with the general technology, the method and the device have the advantages that the clone threads can be matched for a plurality of target clone tasks at the same time, and then the plurality of target clone tasks are processed at the same time, so that the processing efficiency can be improved, the effective utilization of system resources is ensured, and the speed of processing a plurality of snapshot clone tasks is improved.

In an alternative embodiment, as shown in fig. 5 in connection with fig. 4, the process of processing each target cloning task based on the matched cloning thread in S404 may be implemented specifically by the following S501-S502. The following is a detailed description:

s501, the snapshot cloning device determines the cloning stage of the target cloning task based on the matched cloning threads.

In one possible implementation, the snapshot cloning device may determine a cloning stage of a target cloning task processed by a matching cloning thread by looking for the cloning thread, so that the target cloning task is subsequently processed correspondingly based on the cloning stage.

S502, the snapshot cloning device processes each target cloning task based on the cloning stage of each target cloning task.

In one possible implementation, the snapshot cloning device includes a plurality of cloning stages, such as a clone preparation stage, a clone progress stage, and a clone completion stage. The snapshot cloning device performs a stage through cloning preparation firstly, then performs a stage through cloning, and finally performs corresponding processing on the target cloning task in each stage through a cloning completion stage to form a multi-stage processing target cloning task. When the snapshot cloning device processes the target cloning task in error in the three stages, the state of the target cloning task is converted into a cloning failure fail state.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as can be seen from S501-S502, the snapshot cloning device can determine the cloning stage of each target cloning task, can make task management clearer, is convenient to determine the progress and state of the task, and can dynamically allocate resources based on the cloning stage, so that resource waste is avoided.

In an embodiment of the present application, the cloning stage includes: 1. cloning preparation stage, cloning proceeding stage, cloning finishing stage. The process of snapshot cloning the device at different cloning stages is different, and is described in detail below:

1. Cloning preparation stage

In connection with FIG. 5, as shown in FIG. 6, in the cloning preparation stage, the above-mentioned S502 can be specifically realized by the following S601. The following is a detailed description:

s601, when the cloning stage of the target cloning task is a cloning preparation stage, the snapshot cloning device clones the source directory of the target cloning task to the target directory.

In one possible implementation, the clone preparation phase is primarily for preparing for subsequent snapclone. When the cloning stage of the target cloning task is a cloning preparation stage, the snapshot cloning device needs to determine the target path name and clone the source directory of the target cloning task to the target directory.

In an alternative embodiment, as shown in fig. 7 in conjunction with fig. 6, the above S601 may be implemented specifically by the following S701-S702. The following is a detailed description:

s701, the snapshot cloning device acquires a source directory of the target cloning task and directory attributes of the source directory.

Optionally, directory attributes include, but are not limited to: directory type, directory rights, directory owner, directory access time, directory modification time.

S702, the snapshot cloning device creates a target directory in a target file system based on the target path name, and synchronizes the directory attribute of the source directory to the target directory.

Wherein the target path name is used to determine the path of the target directory.

In one possible implementation, the snapclone device may obtain the target path name from the target cloning task, create a target directory under the target path name of the target file system, and then synchronize the directory attribute of the source directory into the target directory.

Illustratively, two directories of/data 1,/data 2 are included in the first file system, and two directories of/test 1,/test 2 are included in the second file system. The snapshot of the source directory/data 1 is stored under the/data 1 directory in the format of snap. The snapshot cloning device obtains a target path name of/test 1 from the target cloning task, creates a catalog/data 1 under/test 1 in the second file system, and determines that the target catalog is/test 1/data1. The snapclone device then synchronizes the directory attributes of the source directory/data 1 to the target directory/test 1/data1.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as known from S701-S702, the snapshot cloning device automatically creates the target directory and synchronizes the directory attribute of the source directory to the target directory, so that consistency of the source directory and the target directory in the attribute can be ensured, and cloning accuracy is ensured.

2. Stage of cloning

In connection with FIG. 5, as shown in FIG. 6, in the cloning preparation stage, the above-mentioned S502 can be specifically realized by the following S602. The following is a detailed description:

s602, when the cloning stage of the target cloning task is a cloning progress stage, the snapshot cloning device clones the directory entry into the target directory based on the type of the directory entry in the source directory.

Optionally, the types of the directory entries include: subdirectories, files.

In one possible implementation, the cloning stage is mainly to clone the directory entry of the source directory into the target directory, i.e. to complete the cloning of the directory sub-tree of the source directory. When the cloning stage of the target cloning task is a cloning proceeding stage, the snapshot cloning device needs to perform different processes according to the type of the directory entry of the source directory, and clone the directory entry into the target directory.

In an alternative embodiment, as shown in fig. 7 in connection with fig. 6, when the type of the directory entry is a subdirectory, S602 may be implemented specifically by the following S703-S704. The following is a detailed description:

s703, the snapshot cloning device acquires the subdirectories in the source directory and the directory attributes of the subdirectories.

Illustratively, the snapclone device obtains the directory attributes including the subdirectory/data 1/dir1 and the subdirectory/data 1/dir1 in the source directory/data 1.

S704, the snapshot cloning device creates a target subdirectory under the target directory, and synchronizes the directory attribute of the subdirectory to the target subdirectory.

Illustratively, the snapclone device creates a subdirectory dir1 under the target directory/test 1/data1 based on the subdirectory/data 1/dir1, determines the target subdirectory/test 1/data1/dir1, and synchronizes the directory attributes of the subdirectory/data 1/dir1 to the target subdirectory/test 1/data1/dir1.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as can be seen from S703-S704, the snapshot cloning device can copy the subdirectory of the source directory into the target subdirectory of the target directory, and can ensure that the target directory and the source directory have the same organization structure, thereby ensuring the cloning accuracy.

In an alternative embodiment, as shown in fig. 7 in connection with fig. 6, when the type of the directory entry is a file, S602 may be implemented specifically by the following S705-S706. The following is a detailed description:

s705, the snapclone device acquires the file in the source directory, and the file data and file attributes of the file.

S706, the snapshot cloning device creates a target file under the target directory, clones file data into the target file, and synchronizes file attributes of the file to the target file.

In one possible implementation, the snapshot cloning device may perform cyclic reading and writing in units of preset bytes when cloning file data into the target file. The snapshot cloning device reads file data in units of preset bytes, and simultaneously writes the data into the target file in units of preset bytes.

Illustratively, the snapshot cloning device creates file flie1 under the target directory/test 1/data1 based on file/data 1/flie1, determines target file/test 1/data1/flie1, and synchronizes file attributes of file/data 1/flie1 to target file/test 1/data1/flie1. When the snapshot cloning device clones file data, the snapshot cloning device reads the file data in 8M units while writing the read data in 8M units into the target file.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as can be seen from S705-S706, when the snapshot cloning device clones a file, both the file data and the attribute data of the file need to be cloned into the target file, so that the integrity of the cloning task and the data consistency between the target file and the file can be ensured, and the reliability of the cloning process is further improved.

3. Cloning completion stage

In connection with FIG. 5, as shown in FIG. 6, in the cloning completion stage, the above-mentioned S502 can be specifically realized by the following S603. The following is a detailed description:

s603, when the cloning stage of the target cloning task is a cloning completion stage, the snapshot cloning device removes the target cloning task from the cloning task queue.

In one possible implementation, when the cloning stage of the target cloning task is a cloning completion stage, the snapshot cloning device needs to remove the target cloning task from the cloning task queue. The snapshot cloning device also needs to delete the record of the target cloning task stored in the database.

The technical scheme provided by the embodiment at least brings the following beneficial effects: from S601-S603, it is known that the snapshot cloning device divides the cloning process into different phases, each with independent tasks, making the cloning process more explicit and easily predictable. When the cloning stage of the target cloning task is the cloning completion stage, the target cloning task is removed from the cloning task queue, so that the target cloning task can be prevented from being executed for multiple times, and the waste of resources is avoided.

In an alternative embodiment, as shown in fig. 8 in connection with fig. 4, the process of determining a clone task queue in S401 may be implemented specifically by the following S801-S802. The following is a detailed description:

S801, the snapshot cloning device acquires snapshot cloning request information.

In one possible implementation, the snapclone device receives the snapclone request information and writes the snapclone request information into a database, from which the snapclone device obtains the snapclone request information.

S802, the snapshot cloning device determines a cloning task based on the snapshot cloning request information, and adds the cloning task to a cloning task queue.

It should be appreciated that the snapshot cloning device needs to process a large amount of snapshot cloning request information, package the snapshot cloning request information into a job cloning task and add the job cloning task to a cloning task queue, and may implement asynchronous processing of the snapshot cloning request information.

In one possible implementation, the snapclone device encapsulates each snapclone request message into one job clone task, and then adds the job task to the clone task queue awaiting execution of the clone task.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as can be seen from S801-S802, the snapshot cloning device adds the cloning task into the task queue, so that the cloning task is effectively managed, the snapshot cloning device can asynchronously process the tasks, the response capability is improved, and the snapshot cloning speed is improved.

In an alternative embodiment, as shown in fig. 8 in conjunction with fig. 4, the snapshot cloning method further includes, prior to determining the cloning task:

s803, the snapshot cloning device receives the snapshot cloning request information, generates an insertion statement based on the snapshot cloning request information, and inserts the insertion statement into a cloning table of the first database.

Wherein, the clone table comprises: snapshot cloning requests information.

Optionally, the snapshot cloning request information includes, but is not limited to: source file system name, source path name, snapshot name, target file system name, target path name, preset cloning time, cloning speed.

In one possible implementation, the user command interface module of the MGR of the snapclone device includes a request distribution module and a database module. The request distribution module receives the snapshot cloning request information and forwards the snapshot cloning request information to the database module. The database module generates a clone table of a first database of the insert statement insert memory schema based on the snapshot clone request information.

Alternatively, the insert statement may be an insert statement in an SQL statement, which is not limited in this application.

Alternatively, the first database may be a sqlite database, which is not limited in this application.

S804, the snapshot cloning device generates an insertion script based on the cloning table, and writes the insertion script into the second database.

The second database is used for storing snapshot cloning request information.

Alternatively, the insertion script may be an SQL statement script, which is not limited in this application.

In one possible implementation, after the snapshot cloning request information is stored in the first database, the data in the cloning table in the first database is traversed by using the command of the backup file, an insertion script is generated, and then the insertion script is written into the second database. Wherein the second database may be a metadata pool in which the insertion script may be written to the object. In this way, persistent storage of snapshot clone request information may be achieved.

Alternatively, the command for backing up the file may be a dump command or an rsync command, which is not limited in this application.

In one possible implementation, when the MGR component in the snapclone device is restarted, the MGR may look up the object from the metadata pool based on the object name of the object, such as snap_db_object. The MGR loads the insertion script from the object content, and rerun generates snapshot cloning request information in the first database.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as known from S803-S804, the snapshot cloning device can intensively store and manage the snapshot cloning request information through the cloning table, and store the snapshot cloning device in the second database, so that the snapshot cloning request information can be stored in a lasting manner, and the safety and reliability of the snapshot cloning request information are ensured.

In an alternative embodiment, in conjunction with fig. 4, as shown in fig. 9, the snapshot cloning method further includes:

s901, the snapshot cloning device obtains the use rate of each disk in the plurality of disks, and determines the disk with the use rate larger than a preset threshold value in the plurality of disks as a target disk.

It should be understood that when the snapshot cloning device performs snapshot cloning, file data are read and written, if the subtree of the source directory is large, directory entries are more, and a large number of inputs and outputs are generated during snapshot cloning, which further affects performance. The snapshot cloning device may specify a preset cloning time for a cloning task at the traffic valley time of the thread, but the cloning task running at the traffic valley time may also have an impact on the device. Since the snapclone is implemented in the MGR component of the snapclone device, the MGR can gather the util usage of all disks in the snapclone device. Therefore, the cloning time interval of each thread in the disk can be adjusted based on the utilization rate of the disk so as to balance the load and reduce the influence on the performance.

In one possible implementation, each OSD service in the snapclone device may manage one disk, and the OSD service may obtain the util index in the disk through the iostat command, and send the util index of the disk to the MGR component. The MGR component determines a target disk of the plurality of disks of the snapshot cloning device for which the utilization exceeds a preset threshold based on the index.

Illustratively, in the case where the preset threshold is 80%, the snapshot cloning device includes 6 disks, the first disk has a utilization of 90%, the second disk has a utilization of 85%, the third disk has a utilization of 80%, the fourth disk has a utilization of 81%, the fifth disk has a utilization of 50%, and the sixth disk has a utilization of 30%. The MGR component may determine that the first disk, the second disk, and the fourth disk are target disks.

S902, the snapshot cloning device determines the proportion of the number of target disks to the number of a plurality of disks.

In one possible implementation, after determining the target disk, the snapclone device determines a ratio of the number of OSDs corresponding to the target disk to the total number of OSDs corresponding to all disks in the device, i.e., a ratio of the number of target disks to the number of multiple disks.

Illustratively, the target disk includes a first disk, a second disk, and a fourth disk, and a total of 3 disks, and the snapshot cloning device includes 6 disks, where the number of the target disks is 50% of the total number of the disks.

S903, the snapshot cloning device adjusts the cloning time interval of each cloning thread in the plurality of magnetic disks based on the proportion.

Wherein the cloning time interval is used to determine the clone thread that matches for each target clone task.

In one possible implementation, the snapshot cloning device may rank the speed of the disk based on scale. And then, based on the speed grade of the disk, correspondingly adjusting the cloning time interval of each thread in the disk, and controlling the speed of disk snapshot cloning by adjusting the cloning time interval.

Illustratively, when the above ratio is less than 40%, the snapshot cloning device determines the speeds of all disks as high-level, and then sets the cloning time interval of each thread of each disk to 0s. When the above ratio is greater than 40% and less than 80%, the snapshot cloning device determines the speeds of all the disks as medium, and then sets the cloning time interval of each thread of each disk to 0.01s. When the above ratio is greater than 80%, the snapshot cloning device determines the speeds of all disks as low-level, and then sets the cloning time interval of each thread of each disk to 0.1s. The snapshot cloning device controls the speed of snapshot cloning by adjusting the cloning time interval for each thread in the disk.

Illustratively, when the first clone task meets a preset condition and there are free first and second clone threads in the pool of clone threads, the snapshot cloning device assigns clone threads to the first clone task based on the clone time interval. The cloning time interval of each cloning thread in the current disk is 0.1s, the time from the first cloning thread to the last processing of the target cloning task is 0.01s, the time from the second cloning thread to the last processing of the target cloning task is 0.15s, and the snapshot cloning device matches the second cloning thread for the first cloning task.

The technical scheme provided by the embodiment at least brings the following beneficial effects: as known from S901 to S903, the snapshot cloning device may obtain the usage rate of the disk, and then adjust the cloning interval of each thread in the disk based on the ratio of the number of disks whose usage rate is greater than the preset threshold to the number of the plurality of disks. The speed of snapshot cloning is controlled by adjusting the cloning time interval of threads in a disk, so that the system load can be balanced, and the system performance can be optimized.

The foregoing description of the solution provided in the embodiments of the present application has been mainly presented in terms of a method. To achieve the above functions, it includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the application may divide the functional modules of the snapclone device according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated modules may be implemented in hardware or in software functional modules. Optionally, the division of the modules in the embodiments of the present application is schematic, which is merely a logic function division, and other division manners may be actually implemented.

Fig. 10 is a schematic structural diagram of a snapclone device according to an embodiment of the present application. The snapshot cloning device can be used to perform the snapshot cloning method shown in fig. 4 to 9. The snapshot cloning device shown in fig. 10 includes: a processing unit 1001.

A processing unit 1001, configured to determine a clone task queue; the cloning task queue comprises at least one cloning task;

the processing unit 1001 is further configured to determine, in the cloning task queue, one or more target cloning tasks that satisfy a preset condition;

the processing unit 1001 is further configured to match a clone thread for each target clone task from the clone thread pool; the clone thread pool comprises at least one clone thread for processing clone tasks;

The processing unit 1001 is further configured to process each target clone task based on the matched clone thread.

Optionally, the processing unit 1001 is specifically configured to: determining a cloning stage of a target cloning task based on the matched cloning threads; each target cloning task is processed based on its cloning stage.

Optionally, the processing unit 1001 is specifically configured to: when the cloning stage of the target cloning task is a cloning preparation stage, cloning a source directory of the target cloning task to a target directory; when the cloning stage of the target cloning task is a cloning progress stage, cloning the directory entry into the target directory based on the type of the directory entry in the source directory; when the cloning stage of the target cloning task is a cloning completion stage, the target cloning task is removed from the cloning task queue.

Optionally, the snapshot cloning device further comprises: a communication unit 1002; the communication unit 1002 is specifically configured to: acquiring a source directory of a target cloning task and directory attributes of the source directory; the processing unit 1001 is specifically configured to: creating a target directory in the target file system based on the target path name, and synchronizing directory attributes of the source directory to the target directory; the target path name is used to determine the path of the target directory.

Optionally, when the type of the directory entry is a subdirectory, the communication unit 1002 is specifically configured to: obtaining subdirectories in a source directory and directory attributes of the subdirectories; the processing unit 1001 is specifically configured to: creating a target subdirectory under the target directory, and synchronizing the directory attributes of the subdirectory to the target subdirectory.

Optionally, when the type of the directory entry is a file, the communication unit 1002 is specifically configured to: acquiring files in a source directory, file data of the files and file attributes; the processing unit 1001 is specifically configured to: creating a target file under the target directory, cloning file data into the target file, and synchronizing file attributes of the file to the target file.

Optionally, the communication unit 1002 is specifically configured to: obtaining snapshot cloning request information; the processing unit 1001 is specifically configured to: based on the snapshot cloning request information, a cloning task is determined and added to a cloning task queue.

Optionally, before determining the clone task queue, the communication unit 1002 is further configured to: receiving snapshot cloning request information; the processing unit 1001 is further configured to: generating an insertion statement based on the snapshot cloning request information, and inserting the insertion statement into a cloning table of the first database; the clone table includes: snapshot cloning request information; the processing unit 1001 is further configured to: generating an insertion script based on the clone table, and writing the insertion script into a second database; the second database is used for storing snapshot cloning request information; wherein obtaining snapshot clone request information includes: and obtaining snapshot cloning request information from the second database.

Optionally, the snapshot cloning device, the communication unit 1002 is further configured to: acquiring the utilization rate of each disk in a plurality of disks; the processing unit 1001 is further configured to: determining a disk with the utilization rate larger than a preset threshold value in the plurality of disks as a target disk; the processing unit 1001 is further configured to: determining the proportion of the number of the target disks to the number of the plurality of disks; the processing unit 1001 is further configured to: adjusting a cloning time interval of each cloning thread in the plurality of disks based on the proportion; the cloning time interval is used to determine the clone thread that matches for each target clone task.

The present application also provides a computer-readable storage medium including computer-executable instructions that, when executed on a computer, cause the computer to perform the snapshot cloning method provided in the above embodiments.

The embodiment of the present application also provides a computer program, which can be directly loaded into a memory and contains software codes, and the computer program can implement the snapshot cloning method provided in the above embodiment after being loaded and executed by a computer.

Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer-readable storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

From the foregoing description of the embodiments, it will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of functional modules is illustrated, and in practical application, the above-described functional allocation may be implemented by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and the division of modules or units, for example, is merely a logical function division, and other manners of division are possible when actually implemented. For example, multiple units or components may be combined or may be integrated into another device, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form. The units described as separate parts may or may not be physically separate, and the parts shown as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units. The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the general technology or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, where the software product includes several instructions to cause a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims (12)

1. A method of snapshot cloning, comprising:

determining a cloning task queue; the cloning task queue comprises at least one cloning task;

determining one or more target cloning tasks meeting preset conditions in the cloning task queue;

matching clone threads for each target clone task from a clone thread pool; the clone thread pool comprises at least one clone thread for processing clone tasks;

and processing each target cloning task based on the matched cloning threads.

2. The method of claim 1, wherein the processing each of the target clone tasks based on the matched clone threads comprises:

determining a cloning stage of the target cloning task based on the matched cloning threads;

and processing each target cloning task based on the cloning stage of each target cloning task.

3. The method of claim 2, wherein said processing each of said target cloning tasks based on said cloning stage comprises:

when the cloning stage of the target cloning task is a cloning preparation stage, cloning a source catalog of the target cloning task to a target catalog;

When the cloning stage of the target cloning task is a cloning progress stage, cloning the directory entry into the target directory based on the type of the directory entry in the source directory;

and when the cloning stage of the target cloning task is a cloning completion stage, removing the target cloning task from the cloning task queue.

4. A method according to claim 3, wherein cloning the source directory of the target cloning task into the target directory comprises:

acquiring a source directory of the target cloning task and directory attributes of the source directory;

creating a target directory in a target file system based on a target path name, and synchronizing directory attributes of the source directory to the target directory; the target path name is used for determining a path of the target directory.

5. The method of claim 3, wherein when the type of the directory entry is a subdirectory, the cloning the directory entry into the target directory based on the type of directory entry in the source directory comprises:

acquiring a subdirectory in the source directory and a directory attribute of the subdirectory;

creating a target subdirectory under the target directory, and synchronizing the directory attribute of the subdirectory to the target subdirectory.

6. A method according to claim 3, wherein when the type of the directory entry is a file, the cloning the directory entry into the target directory based on the type of the directory entry in the source directory comprises:

acquiring files in the source directory, file data of the files and file attributes;

creating a target file under the target directory, cloning the file data into the target file, and synchronizing the file attribute of the file to the target file.

7. The method of any of claims 1-6, wherein determining a clone task queue comprises:

obtaining snapshot cloning request information;

and determining a cloning task based on the snapshot cloning request information, and adding the cloning task to a cloning task queue.

8. The method of claim 7, further comprising, prior to determining the clone task queue:

receiving snapshot cloning request information, generating an insertion statement based on the snapshot cloning request information, and inserting the insertion statement into a cloning table of a first database; the cloning table comprises: the snapshot cloning request information;

Generating an insertion script based on the clone table, and writing the insertion script into a second database; the second database is used for storing the snapshot cloning request information;

the obtaining snapshot cloning request information includes:

and acquiring the snapshot cloning request information from the second database.

9. The method as recited in claim 1, further comprising:

acquiring the utilization rate of each disk in a plurality of disks, and determining the disk with the utilization rate greater than a preset threshold value in the plurality of disks as a target disk;

determining the proportion of the number of the target magnetic disks to the number of a plurality of magnetic disks;

based on the proportion, adjusting the cloning time interval of each cloning thread in a plurality of magnetic disks; the cloning time interval is used to determine a clone thread that matches for each of the target clone tasks.

10. A snapshot cloning device, comprising: a processing unit;

the determining unit is used for determining a cloning task queue; the cloning task queue comprises at least one cloning task;

the determining unit is further used for determining one or more target cloning tasks meeting preset conditions in the cloning task queue;

The determining unit is further configured to match a clone thread for each target clone task from a clone thread pool; the clone thread pool comprises at least one clone thread for processing clone tasks;

the determining unit is further configured to process each target cloning task based on the matched cloning thread.

11. A snapshot cloning device comprising a memory and a processor; the memory is used for storing computer execution instructions, and the processor is connected with the memory through a bus; when the snapshot cloning device is running, the processor executes the computer-executable instructions stored in the memory to cause the snapshot cloning device to perform the snapshot cloning method of any one of claims 1-9.

12. A computer readable storage medium comprising computer executable instructions which, when run on a computer, cause the computer to perform the snapshot cloning method of any of claims 1-9.