CN111897678A - Method, device and processor for improving snapshot data processing speed - Google Patents
Method, device and processor for improving snapshot data processing speed Download PDFInfo
- Publication number
- CN111897678A CN111897678A CN202010732799.9A CN202010732799A CN111897678A CN 111897678 A CN111897678 A CN 111897678A CN 202010732799 A CN202010732799 A CN 202010732799A CN 111897678 A CN111897678 A CN 111897678A
- Authority
- CN
- China
- Prior art keywords
- data
- snapshot
- snapshot data
- pipeline
- protection module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a method, a device and a processor for improving snapshot data processing speed, wherein the method comprises the following steps: acquiring export requirements; creating a first pipeline according to the export demand; acquiring a first Rados block device and a first data protection module; exporting first snapshot data in a first data volume to a first backup end by applying a first Rados block device, a first pipeline and a first data protection module; and/or acquiring an import requirement; creating a second pipeline according to the import requirement; acquiring a second Rados block device and a second data protection module; and importing third snapshot data in the second backup end into a second data volume by applying a second data protection module, a second pipeline and a second Rados block device. According to the scheme, the first pipeline and the second pipeline are used for replacing a hard disk, and the processing speed of snapshot data is improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method, an apparatus, a computer-readable storage medium, and a processor for increasing snapshot data processing speed.
Background
In the existing data protection technology, when processing snapshot data exported by a first Rados block device, a snapshot needs to be cached in a memory or a hard disk, and then the snapshot is read by a first data protection module and correspondingly processed. In order to reduce the consumption of memory or hard disk, the snapshot data is usually divided into a plurality of segments, and the segments are processed separately.
When the snapshot is restored, the second data protection module obtains snapshot data from the second backup terminal, processes the data and writes the data into the local hard disk, and then the second Rados block device imports the snapshot data into the cluster (i.e., the second data volume) from the local to restore the data.
When the existing hard disk is used for caching snapshot data, because the speed of the hard disk is much slower than that of a memory, the speed of the first data protection module for processing the snapshot data is possibly influenced by the speed of the hard disk and possibly influences other services using the hard disk; each segment of the snapshot occupies a large memory and a hard disk, so that the concurrency of the first data protection module cannot be improved, CPU resources are fully utilized, and the efficiency is improved; under the condition of not occupying more memory or hard disk resources, the first data protection module can perform subsequent processing only after the first Rados block device completely derives the snapshot segment, so that the overall processing speed is reduced; the snapshot data exported by the first Rados block device is typically large and occupies a lot of memory or hard disk resources.
The above information disclosed in this background section is only for enhancement of understanding of the background of the technology described herein and, therefore, certain information may be included in the background that does not form the prior art that is already known in this country to a person of ordinary skill in the art.
Disclosure of Invention
The present application mainly aims to provide a method, an apparatus, a computer-readable storage medium, and a processor for increasing a processing speed of snapshot data, so as to solve a problem in the prior art that a processing efficiency of snapshot data is low.
In order to achieve the above object, according to an aspect of the present application, there is provided a method for increasing snapshot data processing speed, including: acquiring export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and data imported into the first backup end is second snapshot data; creating a first pipeline according to the export demand; acquiring a first Rados block device and a first data protection module; exporting the first snapshot data in the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module; and/or acquiring an import requirement, wherein the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and the data imported into the second data volume is fourth snapshot data; creating a second pipeline according to the import requirement; acquiring a second Rados block device and a second data protection module; and importing the third snapshot data in the second backup end into the second data volume by applying the second data protection module, the second pipeline and the second Rados block device.
Optionally, exporting the first snapshot data in the first data volume to the first backup side by applying the first Rados block device, the first pipeline, and the first data protection module, includes: the first Rados block device reads the first snapshot data in the first data volume and sends the first snapshot data to the first pipeline; the first snapshot data is transmitted to the first data protection module through the first pipeline; the first data protection module performs first predetermined processing on the first snapshot data to obtain second snapshot data, and sends the second snapshot data to the first backup end; and the first backup terminal receives the second snapshot data and stores the second snapshot data.
Optionally, applying the second data protection module, the second pipeline, and the second Rados block device, and importing the third snapshot data in the second backup end into the second data volume, where the importing includes: the second data protection module reads out the third snapshot data in the second backup end; the second data protection module performs second predetermined processing on the third snapshot data to obtain fourth snapshot data, and sends the fourth snapshot data to the second pipeline; the fourth snapshot data is transferred to the second Rados block device through the second pipeline; the second Rados block device sends the received fourth snapshot data to the second data volume; the second data volume receives the fourth snapshot data and stores the fourth snapshot data.
Optionally, the first predetermined processing comprises at least one of: encryption processing and compression processing.
Optionally, the second predetermined processing comprises at least one of: decryption processing and decompression processing.
Optionally, there are multiple first Rados block devices, multiple first data protection modules, multiple first pipelines, multiple second Rados block devices, multiple second data protection modules, and multiple second pipelines.
Optionally, the first Rados block device and the first data protection module operate simultaneously, and the second Rados block device and the second data protection module operate simultaneously.
According to another aspect of the present application, there is provided an apparatus for increasing snapshot data processing speed, including: the first obtaining unit is used for obtaining export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and the data imported into the first backup end is second snapshot data; the first creating unit is used for creating a first pipeline according to the export requirement; a second obtaining unit, configured to obtain the first Rados block device and the first data protection module; a exporting unit, configured to export the first snapshot data in the first data volume to the first backup side by applying the first Rados block device, the first pipeline, and the first data protection module; a third obtaining unit, configured to obtain an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data; the second creating unit is used for creating a second pipeline according to the import requirement; a fourth obtaining unit, configured to obtain a second Rados block device and a second data protection module; and an importing unit, configured to apply the second data protection module, the second pipeline, and the second Rados block device, and import the third snapshot data in the second backup end into the second data volume.
According to still another aspect of the present application, a computer-readable storage medium is provided, where the computer-readable storage medium includes a stored program, where the program is executed to control an apparatus where the computer-readable storage medium is located to perform any one of the methods for increasing snapshot data processing speed.
According to still another aspect of the present application, a processor for executing a program is provided, where the program executes any one of the above methods for increasing snapshot data processing speed.
By applying the technical scheme of the application, the first snapshot data is exported from the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module, the second snapshot data is imported from the second backup end to the second data volume by applying the second data protection module, the second pipeline and the second Rados block device, the scheme temporarily stores the first snapshot data through the first pipeline, the second pipeline temporarily stores the second snapshot data, the first snapshot data and the second snapshot data are not required to be cached by a hard disk, the export of the first snapshot data and the import of the second snapshot data are realized, namely, the processing of the snapshot data is not influenced by the processing rate of the hard disk, the first pipeline and the second pipeline only need to occupy less memory resources, the consumption of the memory is reduced, the hard disk resources are not required to be occupied, the processing speed of the snapshot data is improved, and the whole snapshot data processing process does not need to use a hard disk, does not affect the I/O of the hard disk, and reduces the influence on other programs using the hard disk.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 is a flow chart of a method for improving snapshot data processing speed according to an embodiment of the present application;
FIG. 2 is a flow chart of another method for improving snapshot data processing speed according to an embodiment of the present application;
FIG. 3 shows a system diagram for exporting snapshot data according to an embodiment of the present application;
FIG. 4 illustrates a system diagram for importing snapshot data according to an embodiment of the present application; and
fig. 5 is a schematic diagram illustrating an apparatus for increasing snapshot data processing speed according to an embodiment of the present application.
Wherein the figures include the following reference numerals:
01. a first Rados block device; 02. a first data protection module; 03. a first conduit; 04. a second Rados block device; 05. a second data protection module; 06. a second conduit; 100. a first data volume; 200. a first backup terminal; 300. a second data volume; 400. and a second backup terminal.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element or "connected" to the other element through a third element.
For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:
snapshot: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the point in time at which the copy begins). The snapshot may be a copy of the data it represents or may be a replica of the data. The way snapshot data is stored is equivalent to a camera, the data at the point in time is equivalent to a negative film, and the snapshot view is used to present the state of the data at that time, equivalent to a washed out photograph.
Pipeline: the concept invented by Markov Douglas Markov for the Unix command line. A pipe is a series of processes connected by standard input and output, the output of each process being the input to the next process. In practice, it is understood that a small buffer is created between an output process and an input process, and the input process reads data whenever the output process writes the data into the buffer. When the output process is not finished but there is no data write temporarily, the input process enters a waiting state. Compared with the manual cache application, the pipeline implementation is faster and the resource occupation is smaller.
Ceph: ceph is an open-source Distributed storage system based on rados (Reliable, automated Distributed Object Store). The method can combine a plurality of servers into a super-large cluster, integrate disk resources in the machines into a whole to form a large resource pool, and then allocate the resource pool to applications for use according to needs.
Rados Block Device (Rados Block Device, rbd for short): data snapshot segments in the ceph volume may be exported as files via export-diff and import-diff commands, or exported files may be imported into ceph to restore previous data.
Data protection handler (equivalent to first and second data protection modules herein): and a program for processing the snapshot. When the snapshot is exported, operations such as compressing, encrypting, calculating a hash check value and the like can be performed on the snapshot data. When the snapshot is imported, operations such as decompression, decryption, hash value verification and the like can be performed on the obtained data, and finally the original snapshot data is obtained.
As described in the background art, the processing method of snapshot data in the prior art is inefficient, and to solve the above technical problems, embodiments of the present application provide a method, an apparatus, a computer-readable storage medium, and a processor for increasing the processing speed of snapshot data.
According to the embodiment of the application, a method for improving the snapshot data processing speed is provided.
Fig. 1 is a flowchart of a method for increasing snapshot data processing speed according to an embodiment of the present application, and fig. 2 is a flowchart of another method for increasing snapshot data processing speed according to an embodiment of the present application. As shown in fig. 1 and 2, the method comprises the steps of:
step S101, acquiring export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and the data imported into the first backup end is second snapshot data;
step S102, a first pipeline is established according to the export requirement;
step S103, acquiring a first Rados block device and a first data protection module;
step S104, exporting the first snapshot data in the first data volume to the first backup side by using the first Rados block device, the first pipeline, and the first data protection module.
And/or the presence of a gas in the gas,
step S201, acquiring an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data;
step S202, according to the import requirement, a second pipeline is established;
step S203, acquiring a second Rados block device and a second data protection module;
step S204, using the second data protection module, the second pipeline, and the second Rados block device, to import the third snapshot data in the second backup end into the second data volume.
In the scheme, the first snapshot data is exported from the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module, the second snapshot data is imported from the second backup end to the second data volume by applying the second data protection module, the second pipeline and the second Rados block device, the scheme temporarily stores the first snapshot data through the first pipeline, the second pipeline temporarily stores the second snapshot data, the first snapshot data export and the second snapshot data import are realized without using a hard disk to cache the first snapshot data and the second snapshot data, namely, the processing of the snapshot data is not influenced by the processing speed of the hard disk, the first pipeline and the second pipeline only need to occupy less memory resources, the consumption of the memory is reduced, the hard disk resources are not occupied, and the processing speed of the snapshot data is improved, and the whole snapshot data processing process does not need to use a hard disk, does not affect the I/O of the hard disk, and reduces the influence on other programs using the hard disk.
It should be noted that the export process and the import process of the snapshot data may run simultaneously, so as to further increase the processing speed of the snapshot data.
It should be noted that the first data volume and the second data volume may be the same data volume or different data volumes, and the first backup end and the second backup end may be the same backup end or different backup ends.
It should be noted that the first snapshot data and the fourth snapshot data may be the same snapshot data or different snapshot data, and the second snapshot data and the third snapshot data may be the same snapshot data or different snapshot data.
Specifically, the first backup end and the second backup end may be physical devices or virtual devices having a data storage function, such as a cloud backup module and a hard disk.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
In an embodiment of the present application, as shown in fig. 3, the exporting of the first snapshot data in the first data volume 100 to the first backup peer 200 by applying the first Rados block device 01, the first pipeline 03 and the first data protection module 02 includes: the first Rados block device 01 reads the first snapshot data in the first data volume 100 and transmits the first snapshot data to the first pipeline 03; the first snapshot data is transmitted to the first data protection module 02 through the first pipeline 03; the first data protection module 02 performs a first predetermined process on the first snapshot data to obtain the second snapshot data, and sends the second snapshot data to the first backup terminal 200; the first backup end 200 receives the second snapshot data and stores the second snapshot data, the first pipeline 03 serves as a medium for transmitting the first snapshot data, and only a small amount of memory resources are occupied, and compared with a mode in the prior art that the first snapshot data needs to be cached in a memory or a hard disk and then read by the first data protection module 02 and be correspondingly processed when being exported from the first data volume 100, the processing speed of the first data protection module 02 on the data is affected by the hard disk, the scheme does not need to occupy hard disk resources, i.e., the processing speed of the hard disk is not affected, the processing speed of the snapshot data is further increased, and other services using the hard disk are not affected. In addition, through the first pipeline 03, the first Rados block device 01 and the first data protection module 02 can realize instant transmission of snapshot data, and through the second pipeline 06, the second Rados block device 04 and the second data protection module 05 can realize instant transmission of snapshot data, so that the processing speed of the snapshot data is increased.
In another embodiment of the present application, as shown in fig. 4, importing the third snapshot data in the second backup node 400 into the second data volume 300 by using the second data protection module 05, the second pipeline 06, and the second Rados block device 04 includes: the second data protection module 05 reads out the third snapshot data in the second backup terminal 400; the second data protection module 05 performs a second predetermined process on the third snapshot data to obtain the fourth snapshot data, and sends the fourth snapshot data to the second pipeline 06; the fourth snapshot data is transferred to the second Rados block device 04 through the second pipeline 06; the second Rados block device 04 transmitting the received fourth snapshot data to the second data volume 300; the second data volume 300 receives the fourth snapshot data and stores the fourth snapshot data, the second pipeline 06 is used as a medium for transmitting the second snapshot data, and only occupies a small amount of memory resources, and compared with a scheme in the prior art for restoring the third snapshot data to the second data volume 300, after the second data protection module 05 performs the second predetermined processing on the third snapshot data, the obtained fourth snapshot data needs to be temporarily stored in a hard disk, the fourth snapshot data can be imported into the second data volume 300 only by reading the fourth snapshot data from the hard disk by the second Rados block device 04, that is, the third snapshot data is imported into the second data volume 300 and will be influenced by the processing speed of the hard disk, because the data processing speed of the hard disk is much slower than that of the memory, the processing speed of the data will be reduced, and the scheme does not need to occupy the hard disk resources, that is not influenced by the processing speed of the hard disk, the processing speed of the snapshot data is further improved, and other services using the hard disk are not influenced.
In yet another embodiment of the present application, the first predetermined processing includes at least one of: encryption processing and compression processing, that is, when the first snapshot data is exported from the first data volume to the first backup end, the first data protection module is required to encrypt and compress the first snapshot data, so as to facilitate subsequent processing of the second snapshot data.
In an embodiment of the application, the second predetermined processing includes at least one of: and performing decryption and decompression processing, wherein the third snapshot data in the second backup end is encrypted and compressed data, so that the third snapshot data needs to be decrypted and decompressed to restore the third snapshot data to the second data volume.
In a preferred embodiment, there are a plurality of the first Rados block devices, a plurality of the first data protection modules, a plurality of the second Rados block devices, a plurality of the first pipes, a plurality of the second data protection modules, and a plurality of the second pipes, as shown in fig. 3, the first Rados block device 01, the first data protection module 02, and the first pipe 03 form a group, and the first Rados block devices 01, the first data protection modules 02, and the first pipes 03 perform multi-thread processing on the first snapshot data, that is, by opening a plurality of data streams, the rate of exporting the snapshot data is increased; as shown in fig. 4, the one second Rados block device 04, the one second data protection module 05, and the one second pipeline 06 form a group, and the plurality of second Rados block devices 04, the plurality of second data protection modules 05, and the plurality of second pipelines 06 implement multi-thread processing on the third snapshot data, that is, the importing rate of the snapshot data is increased by opening a plurality of data streams. That is, because resources occupied by a single process are reduced, the same snapshot data can be processed by starting more processes simultaneously, and the processing time of each snapshot data is further reduced. And the realization of multiple processes improves the upper limit of the data volume of the snapshot data import and export.
Specifically, the number of the first Rados block devices, the number of the first data protection modules, and the number of the first pipelines may be set according to a data amount of first snapshot data, that is, the number of the first Rados block devices, the number of the first data protection modules, and the number of the first pipelines may be appropriately increased in a case where the data amount of the first snapshot data is large; in the case where the data amount of the first snapshot data is small, the number of the first Rados block devices, the number of the first data protection modules, and the number of the first pipes can be appropriately reduced.
Specifically, the number of the second Rados block devices, the number of the second data protection modules, and the number of the second pipelines may be set according to a data amount of third snapshot data, that is, the number of the second Rados block devices, the number of the second data protection modules, and the number of the second pipelines may be appropriately increased in a case where the data amount of the third snapshot data is large; in the case where the data amount of the third snapshot data is small, the number of the second Rados block devices, the number of the above-described second data protection modules, and the number of the second pipelines can be appropriately reduced.
In another preferred embodiment, as shown in fig. 3 and 4, since both ends of the first pipeline 03 are configured to synchronously transmit and receive data and the memory consumed by the first pipeline 03 is almost 0, the first Rados block device 01 and the first data protection module 02 may operate simultaneously, and since both ends of the second pipeline 06 are configured to synchronously transmit and receive data and the memory consumed by the second pipeline 06 is almost 0, the second Rados block device 04 and the second data protection module 05 may operate simultaneously, so that snapshot data is rapidly transmitted from the first Rados block device 01 to the first data protection module 02 and snapshot data is rapidly transmitted from the second data protection module 05 to the second Rados block device 04; that is, while the first Rados block device 01 processes the first snapshot data, the first data protection module 02 may process the first snapshot data, that is, the first Rados block device 01 may send the received first snapshot data to the first pipeline 03 every predetermined time period, the first data protection module 02 receives the first snapshot data transmitted through the first pipeline 03 and then processes the first snapshot data in time, each data transmission can be realized only by occupying a smaller memory, that is, under the condition of not occupying more memory or hard disk resources, the first data protection module 02 does not need to wait until all the first snapshot data are completely exported to the first data protection module 02 and then process the first snapshot data, and the data volume of the snapshot data exported by the first Rados block device 01 is smaller and does not occupy more memory or hard disk resources; similarly, the second Rados block device 04 and the second data protection module 05 can also run simultaneously, so that the processing speed of the snapshot data is increased on the premise of not occupying much memory or hard disk resources. In addition, the first Rados block device 01 and the first data protection module 02 are simultaneously executed through the first pipeline 03, and the second Rados block device 04 and the second data protection module 05 are simultaneously executed through the second pipeline 06, so that the utilization rate of the CPU is improved. And by improving the data processing efficiency and reducing the backup time, the situation of backup failure caused by network and machine faults can be reduced.
The embodiment of the present application further provides a device for increasing the processing speed of snapshot data, and it should be noted that the device for increasing the processing speed of snapshot data in the embodiment of the present application may be used to execute the method for increasing the processing speed of snapshot data provided in the embodiment of the present application. The following describes an apparatus for improving snapshot data processing speed according to an embodiment of the present application.
Fig. 5 is a schematic diagram of an apparatus for increasing snapshot data processing speed according to an embodiment of the present application. As shown in fig. 5, the apparatus includes:
a first obtaining unit 10, configured to obtain an export demand, where the export demand is a demand for exporting first snapshot data in a first data volume to a first backup side, and data imported into the first backup side is second snapshot data;
a first creating unit 20, configured to create a first pipeline according to the export requirement;
a second obtaining unit 30, configured to obtain the first Rados block device and the first data protection module;
an export unit 40, configured to export the first snapshot data in the first data volume to the first backup side by applying the first Rados block device, the first pipeline, and the first data protection module;
a third obtaining unit 50, configured to obtain an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data;
a second creating unit 60, configured to create a second pipeline according to the import requirement;
a fourth obtaining unit 70, configured to obtain a second Rados block device and a second data protection module;
an importing unit 80, configured to import the third snapshot data in the second backup end into the second data volume by applying the second data protection module, the second pipeline, and the second Rados block device.
In the scheme, the export unit exports the first snapshot data from the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module, the import unit imports the second snapshot data from the second backup end to the second data volume by applying the second data protection module, the second pipeline and the second Rados block device, the scheme temporarily stores the first snapshot data and the second snapshot data by the first pipeline, the second pipeline temporarily stores the second snapshot data, the first snapshot data and the second snapshot data are not required to be cached by a hard disk, the export of the first snapshot data and the import of the second snapshot data are realized, namely, the processing of the snapshot data is not influenced by the processing rate of the hard disk, the first pipeline and the second pipeline only need to occupy less memory resources, the consumption of the memory is reduced, the hard disk resources are not required to be occupied, and the processing speed of the snapshot data is improved, and the whole snapshot data processing process does not need to use a hard disk, does not affect the I/O of the hard disk, and reduces the influence on other programs using the hard disk.
It should be noted that the export process and the import process of the snapshot data may run simultaneously, so as to further increase the processing speed of the snapshot data.
It should be noted that the first data volume and the second data volume may be the same data volume or different data volumes, and the first backup end and the second backup end may be the same backup end or different backup ends.
It should be noted that the first snapshot data and the fourth snapshot data may be the same snapshot data or different snapshot data, and the second snapshot data and the third snapshot data may be the same snapshot data or different snapshot data.
Specifically, the first backup end and the second backup end may be physical devices or virtual devices having a data storage function, such as a cloud backup module and a hard disk.
In an embodiment of the present application, as shown in fig. 3, the first Rados block device 01 includes a first read module and a first write module, the first pipeline 03 includes a first transmission module, the first data protection module 02 includes a first predetermined processing module, a second read module and a second write module, the first backup side 200 includes a first receiving module and a first storage module, the first read module is configured to read the first snapshot data in the first data volume 100, and the first write module is configured to send the first snapshot data to the first pipeline 03; the first transmission module is used for transmitting first snapshot data from the first Rados block device 01 to the first data protection module 02, the second reading module is used for reading the first snapshot data into the first data protection module 02, the first predetermined processing module is used for performing first predetermined processing on the first snapshot data to obtain the second snapshot data, the second writing module is used for sending the second snapshot data to the first backup terminal 200, the first receiving module is used for receiving the second snapshot data, and the first storage module is used for storing the second snapshot data; compared with the prior art in which the first snapshot data is exported from the first data volume 100, the first pipeline 03 only occupies a small amount of memory resources, and needs to cache the first snapshot data in the memory or the hard disk, and then read and perform corresponding processing by the first data protection module 02, that is, the processing speed of the first data protection module 02 on the data is affected by the hard disk, and the scheme does not need to occupy hard disk resources, that is, the processing speed of the hard disk is not affected, so that the processing speed of the snapshot data is further increased, and other services using the hard disk are not affected. In addition, through the first pipeline 03, the first Rados block device 01 and the first data protection module 02 can realize instant transmission of snapshot data, and through the second pipeline 06, the second Rados block device 04 and the second data protection module 05 can realize instant transmission of snapshot data, so that the processing speed of the snapshot data is increased.
In another embodiment of the present application, as shown in fig. 4, the second data protection module 05 includes a third read module, a first predetermined processing module, and a third write module, the second pipeline 06 includes a second transmission module, the second Rados block device 04 includes a fourth read module and a fourth write module, the second data volume 300 includes a second receiving module and a second storage module, and the third read module is configured to read out the third snapshot data in the second backup terminal 400; the first scheduled processing module is used for performing second scheduled processing on the third snapshot data to obtain the fourth snapshot data; the third writing module is configured to send the fourth snapshot data to the second pipeline 06; the second transmission module is configured to transmit the fourth snapshot data from the second data protection module 05 to the second Rados block device 04; the fourth reading module is configured to read out fourth snapshot data from the second pipeline 06, the fourth writing module is configured to write the fourth snapshot data into the second data volume 300, the second receiving module is configured to receive the fourth snapshot data, and the second storage module is configured to store the fourth snapshot data; the second pipeline 06, as a medium for transmitting the second snapshot data, only occupies a relatively small amount of memory resources, and compared with the scheme in the prior art that restores the third snapshot data to the second data volume 300, the second data protection module 05 needs to temporarily store the obtained fourth snapshot data in the hard disk after performing the second predetermined processing on the third snapshot data, the fourth snapshot data is read from the hard disk by the second Rados block device 04 before it can be imported into the second data volume 300, that is, the third snapshot data is imported into the second data volume 300, which is affected by the processing speed of the hard disk, since the data processing speed of the hard disk is much slower than that of the memory, the processing speed of the data can be reduced, the scheme does not need to occupy hard disk resources, namely, the scheme is not influenced by the processing speed of the hard disk, the processing speed of the snapshot data is further improved, and other services using the hard disk are not influenced.
The device for improving the processing speed of the snapshot data comprises a processor and a memory, wherein the first acquiring unit, the first creating unit, the second acquiring unit, the exporting unit, the third acquiring unit, the second creating unit, the fourth acquiring unit, the importing unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more, and the processing speed of the snapshot data is improved by adjusting the kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
The embodiment of the invention provides a computer-readable storage medium, which comprises a stored program, wherein when the program runs, a device where the computer-readable storage medium is located is controlled to execute the method for improving the processing speed of snapshot data.
The embodiment of the invention provides a processor, which is used for running a program, wherein the method for improving the processing speed of snapshot data is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein when the processor executes the program, at least the following steps are realized:
step S101, acquiring export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and the data imported into the first backup end is second snapshot data;
step S102, a first pipeline is established according to the export requirement;
step S103, acquiring a first Rados block device and a first data protection module;
step S104, exporting the first snapshot data in the first data volume to the first backup side by using the first Rados block device, the first pipeline, and the first data protection module;
and/or the presence of a gas in the gas,
step S201, acquiring an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data;
step S202, according to the import requirement, a second pipeline is established;
step S203, acquiring a second Rados block device and a second data protection module;
step S204, using the second data protection module, the second pipeline, and the second Rados block device, to import the third snapshot data in the second backup end into the second data volume.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program of initializing at least the following method steps when executed on a data processing device:
step S101, acquiring export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and the data imported into the first backup end is second snapshot data;
step S102, a first pipeline is established according to the export requirement;
step S103, acquiring a first Rados block device and a first data protection module;
step S104, exporting the first snapshot data in the first data volume to the first backup side by using the first Rados block device, the first pipeline, and the first data protection module;
and/or the presence of a gas in the gas,
step S201, acquiring an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data;
step S202, according to the import requirement, a second pipeline is established;
step S203, acquiring a second Rados block device and a second data protection module;
step S204, using the second data protection module, the second pipeline, and the second Rados block device, to import the third snapshot data in the second backup end into the second data volume.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
From the above description, it can be seen that the above-described embodiments of the present application achieve the following technical effects:
1) the method for improving the processing speed of the snapshot data exports the first snapshot data from the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module, and imports the second snapshot data from the second backup end to the second data volume by applying the second data protection module, the second pipeline and the second Rados block device, the scheme temporarily stores the first snapshot data through the first pipeline, the second pipeline temporarily stores the second snapshot data, the first snapshot data and the second snapshot data are not required to be cached by a hard disk, the exporting of the first snapshot data and the importing of the second snapshot data are realized, namely the processing of the snapshot data is not influenced by the processing speed of the hard disk, the first pipeline and the second pipeline only need to occupy less memory resources, the memory consumption is reduced, the hard disk resources are not required to be occupied, the processing speed of the snapshot data is improved, the hard disk is not needed in the whole snapshot data processing process, the influence on the I/O of the hard disk is avoided, and the influence on other programs using the hard disk is reduced.
2) The device for improving the processing speed of the snapshot data comprises a leading-out unit, an importing unit, a second pipeline and a second Rados block device, wherein the leading-out unit is used for leading out the first snapshot data from a first data volume to a first backup end, the importing unit is used for leading the second snapshot data from a second backup end to a second data volume by using the second data protection module, the second pipeline and the second Rados block device, the scheme temporarily stores the first snapshot data through the first pipeline, the second pipeline temporarily stores the second snapshot data, the first snapshot data and the second snapshot data are cached without using a hard disk, the leading-out of the first snapshot data and the importing of the second snapshot data are realized, namely the processing of the snapshot data is not influenced by the processing speed of the hard disk, the first pipeline and the second pipeline only need to occupy less memory resources, the consumption of a memory is reduced, and the hard disk resources are not occupied, the processing speed of the snapshot data is improved, the hard disk is not needed in the whole snapshot data processing process, the influence on the I/O of the hard disk is avoided, and the influence on other programs using the hard disk is reduced.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for improving snapshot data processing speed is characterized by comprising the following steps:
acquiring export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and data imported into the first backup end is second snapshot data;
creating a first pipeline according to the export demand;
acquiring a first Rados block device and a first data protection module;
exporting the first snapshot data in the first data volume to the first backup end by applying the first Rados block device, the first pipeline and the first data protection module;
and/or the presence of a gas in the gas,
acquiring an import requirement, wherein the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and the data imported into the second data volume is fourth snapshot data;
creating a second pipeline according to the import requirement;
acquiring a second Rados block device and a second data protection module;
and importing the third snapshot data in the second backup end into the second data volume by applying the second data protection module, the second pipeline and the second Rados block device.
2. The method of claim 1, wherein exporting the first snapshot data in the first data volume to the first backup peer using the first Rados block device, the first pipeline, and the first data protection module comprises:
the first Rados block device reads the first snapshot data in the first data volume and sends the first snapshot data to the first pipeline;
the first snapshot data is transmitted to the first data protection module through the first pipeline;
the first data protection module performs first predetermined processing on the first snapshot data to obtain second snapshot data, and sends the second snapshot data to the first backup end;
and the first backup terminal receives the second snapshot data and stores the second snapshot data.
3. The method of claim 1, wherein importing the third snapshot data in the second backup peer to the second data volume using the second data protection module, the second pipeline, and the second Rados block device comprises:
the second data protection module reads out the third snapshot data in the second backup end;
the second data protection module performs second predetermined processing on the third snapshot data to obtain fourth snapshot data, and sends the fourth snapshot data to the second pipeline;
the fourth snapshot data is transferred to the second Rados block device through the second pipeline;
the second Rados block device sends the received fourth snapshot data to the second data volume;
the second data volume receives the fourth snapshot data and stores the fourth snapshot data.
4. The method of claim 2, wherein the first predetermined process comprises at least one of:
encryption processing and compression processing.
5. The method of claim 3, wherein the second predetermined process comprises at least one of:
decryption processing and decompression processing.
6. The method of any of claims 1-5, wherein there are a plurality of said first Rados block devices, a plurality of said first data protection modules, a plurality of said first pipes, a plurality of said second Rados block devices, a plurality of said second data protection modules, and a plurality of said second pipes.
7. The method of any of claims 1-5, wherein the first Rados block device and the first data protection module run concurrently, and wherein the second Rados block device and the second data protection module run concurrently.
8. An apparatus for increasing snapshot data processing speed, comprising:
the first obtaining unit is used for obtaining export requirements, wherein the export requirements are requirements for exporting first snapshot data in a first data volume to a first backup end, and the data imported into the first backup end is second snapshot data;
the first creating unit is used for creating a first pipeline according to the export requirement;
a second obtaining unit, configured to obtain the first Rados block device and the first data protection module;
a exporting unit, configured to export the first snapshot data in the first data volume to the first backup side by applying the first Rados block device, the first pipeline, and the first data protection module;
a third obtaining unit, configured to obtain an import requirement, where the import requirement is a requirement for importing third snapshot data in a second backup end into a second data volume, and data imported into the second data volume is fourth snapshot data;
the second creating unit is used for creating a second pipeline according to the import requirement;
a fourth obtaining unit, configured to obtain a second Rados block device and a second data protection module;
and an importing unit, configured to apply the second data protection module, the second pipeline, and the second Rados block device, and import the third snapshot data in the second backup end into the second data volume.
9. A computer-readable storage medium, comprising a stored program, wherein when the program runs, the computer-readable storage medium is controlled to execute a method for improving snapshot data processing speed according to any one of claims 1 to 7.
10. A processor, characterized in that the processor is configured to execute a program, wherein the program executes the method for increasing the processing speed of snapshot data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010732799.9A CN111897678B (en) | 2020-07-27 | 2020-07-27 | Method, device and processor for improving snapshot data processing speed |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010732799.9A CN111897678B (en) | 2020-07-27 | 2020-07-27 | Method, device and processor for improving snapshot data processing speed |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111897678A true CN111897678A (en) | 2020-11-06 |
| CN111897678B CN111897678B (en) | 2023-10-03 |
Family
ID=73190139
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010732799.9A Active CN111897678B (en) | 2020-07-27 | 2020-07-27 | Method, device and processor for improving snapshot data processing speed |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111897678B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050216788A1 (en) * | 2002-11-20 | 2005-09-29 | Filesx Ltd. | Fast backup storage and fast recovery of data (FBSRD) |
| CN103229171A (en) * | 2010-11-30 | 2013-07-31 | 国际商业机器公司 | Snapshot based replication |
| CN109324874A (en) * | 2018-09-25 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of virutal machine memory snapshot imports the method, system and device of block device |
| US20200057697A1 (en) * | 2017-08-07 | 2020-02-20 | Datto, Inc. | Multiclient Backup Replication Apparatuses, Methods and Systems |
| US20200201831A1 (en) * | 2018-12-21 | 2020-06-25 | Palantir Technologies Inc. | Data pipeline branching |
| CN111399774A (en) * | 2020-03-09 | 2020-07-10 | 星辰天合（北京）数据科技有限公司 | Data processing method and device based on snapshot under distributed storage system |
-
2020
- 2020-07-27 CN CN202010732799.9A patent/CN111897678B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050216788A1 (en) * | 2002-11-20 | 2005-09-29 | Filesx Ltd. | Fast backup storage and fast recovery of data (FBSRD) |
| CN103229171A (en) * | 2010-11-30 | 2013-07-31 | 国际商业机器公司 | Snapshot based replication |
| US20200057697A1 (en) * | 2017-08-07 | 2020-02-20 | Datto, Inc. | Multiclient Backup Replication Apparatuses, Methods and Systems |
| CN109324874A (en) * | 2018-09-25 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of virutal machine memory snapshot imports the method, system and device of block device |
| US20200201831A1 (en) * | 2018-12-21 | 2020-06-25 | Palantir Technologies Inc. | Data pipeline branching |
| CN111399774A (en) * | 2020-03-09 | 2020-07-10 | 星辰天合（北京）数据科技有限公司 | Data processing method and device based on snapshot under distributed storage system |
Non-Patent Citations (2)
| Title |
|---|
| WEIJUN XIAO 等: "Design and Analysis of Block-Level Snapshots for Data Protection and Recovery", 《IEEE TRANSACTIONS ON COMPUTERS》, pages 1615 - 1625 * |
| 徐茂红 等: "基于大数据量的Redis技术应用与研究", 《信息技术与信息化》, pages 228 - 230 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111897678B (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10545823B2 (en) | Accelerating erasure code replication in distributed systems | |
| CN109791471B (en) | Virtualizing non-volatile storage at a peripheral device | |
| US9405483B1 (en) | Optimized write performance at block-based storage during volume snapshot operations | |
| US20120102455A1 (en) | System and apparatus for hosting applications on a storage array via an application integration framework | |
| CN108829508B (en) | Task processing method and device | |
| CN113079200A (en) | Data processing method, device and system | |
| US9420070B2 (en) | Streaming zip | |
| CN110018878B (en) | Distributed system data loading method and device | |
| CN108845876B (en) | Service distribution method and device | |
| CN109324874B (en) | Method, system and device for importing virtual machine memory snapshot into block device | |
| CN111708738A (en) | Method and system for realizing data inter-access between hdfs of hadoop file system and s3 of object storage | |
| CN113411404A (en) | File downloading method, device, server and storage medium | |
| CN104063284B (en) | The method and apparatus running application program | |
| CN115390991A (en) | Virtual machine import and export method, device, equipment and storage medium | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| CN115470156A (en) | RDMA-based memory use method, system, electronic device and storage medium | |
| CN111294377B (en) | Dependency network request sending method, terminal device and storage medium | |
| CN113411363A (en) | Uploading method of image file, related equipment and computer storage medium | |
| CN106383667A (en) | Method and apparatus for LUN migration in business operation | |
| CN108111630B (en) | Zookeeper cluster system and connection method and system thereof | |
| CN111897678B (en) | Method, device and processor for improving snapshot data processing speed | |
| CN110764930A (en) | Request or response processing method and device based on message mode | |
| CN111274176B (en) | Information processing method, electronic equipment, system and storage medium | |
| CN110633564B (en) | File generation method and device | |
| CN109561123B (en) | Token caching method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100094 101, floors 1-5, building 7, courtyard 3, fengxiu Middle Road, Haidian District, Beijing Applicant after: Beijing Xingchen Tianhe Technology Co.,Ltd. Address before: 100097 room 806-1, block B, zone 2, Jinyuan times shopping center, indigo factory, Haidian District, Beijing Applicant before: XSKY BEIJING DATA TECHNOLOGY Corp.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |