CN112115001A

CN112115001A - Data backup method and device, computer storage medium and electronic equipment

Info

Publication number: CN112115001A
Application number: CN202010990027.5A
Authority: CN
Inventors: 唐之享
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Priority date: 2020-09-18
Filing date: 2020-09-18
Publication date: 2020-12-22
Anticipated expiration: 2040-09-18
Also published as: CN112115001B

Abstract

The application discloses a data backup method, a data backup device, a computer storage medium and electronic equipment, wherein the method comprises the following steps: writing the data to be backed up into R copies; wherein each copy comprises N data blocks; after the data to be backed up is written, calculating check data according to the N data blocks in the target copy, and writing the check data into the M check blocks; the M check blocks and the data blocks of the R copies are stored in the same stripe; r is a positive integer greater than or equal to 2, and N and M are positive integers; deleting data in copies of the strip other than the target copy. By implementing the embodiment of the application, the bandwidth waste can be avoided, and the method is compatible with the reading and writing of the small files.

Description

Data backup method and device, computer storage medium and electronic equipment

Technical Field

The present application relates to the field of cloud storage, and in particular, to a data backup method and apparatus, a computer storage medium, and an electronic device.

Background

With the continuous development of information technology, cloud storage systems hold a great importance. In order to prevent data loss and influence on user use, a multi-copy mode is usually adopted to implement data redundancy backup. With the increasing amount of data generated by the internet, the cost of implementing data redundancy backup by using a multi-copy mode is higher and higher. The mainstream cloud storage system adopts an erasure code mode to realize redundant backup. Current erasure code schemes include both continuous and bar layouts. The continuous layout can cause bandwidth waste, and the bar layout is not compatible with reading and writing of small files.

Disclosure of Invention

The embodiment of the application provides a data backup method and device, a computer storage medium and electronic equipment, which can avoid bandwidth waste and are compatible with small file reading and writing.

In a first aspect, an embodiment of the present application provides a data backup method, including:

writing the data to be backed up into R copies; wherein each copy comprises N data blocks;

after the data to be backed up is written, calculating check data according to the N data blocks in the target copy, and writing the check data into the M check blocks; the M check blocks and the data blocks of the R copies are stored in the same stripe; r is a positive integer greater than or equal to 2, and N and M are positive integers;

deleting data in copies of the stripe other than the target copy

In a second aspect, an embodiment of the present application provides a data backup apparatus, including:

the backup module is used for writing the data to be backed up into the R copies; wherein each copy comprises N data blocks;

the checking module is used for calculating checking data according to the N data blocks in the target copy after the data to be backed up is written in, and writing the checking data into the M checking blocks; the M check blocks and the data blocks of the R copies are stored in the same stripe; r is a positive integer greater than or equal to 2, and N and M are positive integers;

and the deleting module is used for deleting the data in other copies except the target copy in the strip.

In a third aspect, an embodiment of the present application provides a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps provided in the first aspect of the embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps as provided by the first aspect of the embodiments of the present application.

The beneficial effects brought by the technical scheme provided by some embodiments of the application at least comprise:

according to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the check data is calculated according to the target copy, the check data and the data contained in each copy are stored in the same strip, and the data in the other copies except the target copy in the strip are deleted after the calculation of the check data is completed. By implementing the embodiment of the application, the data block storage space of each strip can be pre-allocated in advance, the integration of a multi-copy system and an erasure code system is realized, the operation and maintenance of the cloud storage system for unified resource management are facilitated, the data is prevented from being migrated from the multi-copy system to the erasure code system, the transmission bandwidth is saved, the cost of the cloud storage system is greatly saved, the small file can be read and written compatibly, and the delay of uploading and downloading of the small file is prevented from being too large.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a data backup system according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a cloud storage system with a continuous layout according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a cloud storage system with a stripe layout according to an embodiment of the present application;

fig. 4 is a schematic flowchart of a data backup method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a cloud storage system according to an embodiment of the present application;

fig. 6 is a schematic flowchart of another data backup method according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a data backup device according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.

In the description of the present application, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art. Further, in the description of the present application, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Fig. 1 schematically illustrates a structural diagram of a data backup system provided in an embodiment of the present application. As shown in fig. 1, the data backup system may include at least: a control terminal 110 and a plurality of storage terminals (120a, 120b, 120 c). The number of the control terminals 110 and the number of the storage terminals are not limited to the number shown in the figure. The number of control terminals may be plural, and the number of storage terminals may be more or less. The number of the control terminals and the number of the storage terminals are not limited in the embodiment of the application.

The control terminal 110 is used to control which storage terminal or terminals the data to be stored is stored in, which disk of a certain storage terminal is stored, and the like. The control terminal can also be used for setting the sizes of the data blocks and the check blocks in the storage terminal, the sealing conditions of the data blocks and the like. After the data storage is finished, the control terminal can also be used for issuing an instruction for deleting the redundant copies.

The storage terminal is used for receiving the control instruction of the control terminal and writing the data to be stored into a specific disk. The storage terminal is also used for receiving an instruction for setting the size of the data block and an instruction for sealing the database by the control terminal, and sealing the data block under the condition that the data block meets the sealing condition. The storage terminal is also used for receiving a deletion instruction of the control terminal, and more copies can be deleted after the data backup is finished.

The control terminal and the storage terminal related to the embodiment of the present application may be a mobile phone, a tablet Computer, a desktop Computer, a laptop Computer, a notebook Computer, an Ultra-mobile Personal Computer (UMPC), a handheld Computer, a netbook, a Personal Digital Assistant (PDA), a server, and the like.

The data backup method provided in the following embodiments of the present application is executed by the control terminal.

The erasure coding technique according to the embodiment of the present application will be described next. The most common code of the erasure code technology is an RS (Reed-Solomon) code, and the RS code is realized by equally dividing an original data block into N data blocks, then selecting a proper check matrix, generally adopting a reversible Cauchy matrix, generating M check block data in a matrix operation mode, and respectively writing the N + M check block data into different storage nodes, so that the data damage of the most M storage nodes can be tolerated, and the original data can be recovered through the matrix operation.

To ensure that the backup data is reliable, the data is typically replicated in multiple copies, with the data for each copy being the same. And storing the data of each copy in a plurality of data blocks, wherein the data blocks are a plurality of allocated memory slice areas in the same stripe (stripe).

The cloud storage system with continuous layout provided by the embodiment of the application is introduced next. The cloud storage system can adopt a multi-copy mode to realize redundant backup of large file data. Specifically, the original data (data to be backed up) may be written into a plurality of large files, erasure code calculation is performed on data blocks of a specific size from the large files each time to obtain a check block, and finally, the data blocks and the check block are written into an erasure code system together.

Fig. 2 schematically illustrates a structural diagram of a continuously-arranged cloud storage system provided by an embodiment of the present application. As shown in fig. 2, the cloud storage system is a three-copy cloud storage system. The original data (data to be backed up) may first be written to a large file in a three-copy system and then migrated to the erasure coding system. The erasure code system adopts an N4M2 mode, wherein N represents that the number of the databases is 4, and M represents that the data of the check block is 2. If the size of each data block in one stripe is 128M, 128 x 4 to 512M data is obtained from a large file in each migration, then the data blocks are divided into 4 data blocks (D1, D2, D3 and D4) and 2 check blocks (P1 and P2) are generated through RS codes, and finally the data blocks and the check blocks are written into an erasure code system together.

Next, a cloud storage system with a bar-shaped layout provided in an embodiment of the present application is described. And (3) in a bar-type layout mode, each uploaded data is equally divided into data blocks with the same size, 0 is supplemented behind the data which cannot be equally divided, and the RS codes are adopted to calculate the check blocks in real time. These same size data blocks and parity blocks together form a stripe, and the data blocks and parity blocks of the same stripe are written to different storage nodes.

Fig. 3 shows a schematic structural diagram of a cloud storage system with a bar-shaped layout according to an embodiment of the present application. As shown in fig. 3, the cloud storage system adopts an N4M2 mode, 4M original data (data to be backed up) can be divided into 4 data blocks (D1, D2, D3, D4), 2 check blocks (P1, P2) are generated by real-time calculation, and 6 data blocks are written into different storage nodes. The generated data blocks are not of the same size for different sizes of raw data.

The following describes a data backup method provided by an embodiment of the present application, which can be used for redundant backup of data. As shown in fig. 4, the data backup method may include the following steps:

s401: and writing the data to be backed up into the R copies.

Wherein each copy comprises N data blocks. The capacity of each data block is the same size.

Referring to fig. 5, the following illustrates a process of allocating stripes in an R3N4M2 mode, where R denotes that the number of copies in the multi-copy mode is 3, N denotes that the number of data blocks in the erasure code mode is 4, and M denotes that the number of parity blocks in the erasure code mode is 2. Block0-block13 are allocated to one stripe at a time to form 14 data blocks, wherein block0, block4 and block8 store completely identical data and form 3 copy groups, and except for the last row, the first three rows and each column are three copies. Wherein 4 data blocks of block0, block1, block2 and block3 are stored in the first copy.

And the R copies are respectively stored in different servers and/or different storage spaces of the same server.

Possibly, each copy can be stored in a different server, ensuring that data reading and writing are not affected when one of the servers is broken.

Possibly, each copy may also be stored in a different storage space of the same server, and data security may also be guaranteed when the number of devices is limited.

Possibly, partial copies may be stored in different servers, and the remaining partial copies may be stored in different storage spaces of the same server.

S402: after the data to be backed up is written, check data is calculated according to the N data blocks in the target copy, and the check data is written into the M check blocks.

Wherein, M check blocks and R copy data blocks are stored in the same stripe. R is a positive integer greater than or equal to 2, and N and M are positive integers. Exemplarily, the present embodiment may include 3 copies, and the writing is completed after the backup data is written into the 3 copies respectively. And selecting one copy from the 3 copies as a target copy to calculate check data. The storage space (check block) for the check data is a pre-allocated storage space. After the check data is calculated, the check data is directly written into the check block.

S403: deleting data in copies of the strip other than the target copy.

Specifically, if one copy is selected as the target copy in S402 to calculate the check block, after the calculation of the check block is completed, the data in the other copies may be deleted to release the storage space. For example, reference may be made again to fig. 5. If the 3 rd copy is selected as the target copy, the correction data is calculated according to the data blocks block8-block11 in the 3 rd copy, and after the correction data is written into the blocks 12-13, the data in the 1 st copy and the data in the second copy in the stripe can be deleted, namely the data in the blocks 0-7 are deleted.

The embodiment of the application provides another data backup method. As shown in fig. 6, the data backup method may include the following steps:

s601: the size of the capacity of each data block is set.

Specifically, the size of the capacity of the data block is set by the control terminal in a unified manner. Illustratively, the size of the data block is, for example, but not limited to, 128M.

S602: and writing the data to be backed up into the R copies according to the capacity of the data block.

Specifically, the data to be backed up is written into the first data block of each of the R copies according to the capacity size of the data block. And when the first data block meets the sealing condition, writing the remaining data to be backed up into the next data block of each copy until the data to be backed up is completely written.

Specifically, the capacity size of each block is set in advance, for example, the capacity of the block is set to 128M. Block blocks are allocated to each stripe, for example, R3N4M2, 14 blocks need to be allocated, and the actual writable capacity size of the multiple copy blocks in each stripe is 4 × 128 — 512M capacity. And (3) simultaneously writing 3 blocks in each object file, and returning success only after 3 blocks are successfully written at the same time, for example, according to an R3N4M2 model, simultaneously writing

blocks

0, 4 and 8, after writing, simultaneously writing

blocks

1, 5 and 9, and so on until all data blocks are written. In the same copy, a block is written full and then the next block is written, for example, taking the first copy as an example, the writing sequence is from block0 to block3, block0 is written first, and block1 is written after block0 is full until the copy meets the blocking condition.

It will be appreciated that the write speed of each copy is not necessarily exactly the same, and there may be some nodes that fail, resulting in failed writes or delays to the copies stored in the node.

Specifically, the sealing condition of the data block may be that the remaining capacity of the data block is lower than a preset threshold. Wherein, the threshold is also preset, for example but not limited to 128 kb.

S603: and after the data to be backed up is written, determining a target copy according to a preset rule.

Specifically, the determining of the target copy according to the preset rule may be, for example, selecting a copy in which data is successfully written as the target copy according to a default copy arrangement order.

For example, the default copy arrangement may be copy 1, copy 2, copy 3. If the 1 st copy fails to write data in the process of writing data, and the 2 nd copy and the 3 rd copy succeed in writing data, a target copy can be selected from the 2 nd copy and the 3 rd copy. As can be seen from the default ranking, copy No. 2 takes precedence over copy No. 3, and copy No. 2 can be selected as the target copy at this time.

It should be understood that the above default arrangement order of the copies is only an exemplary order, and other arrangement orders of the copies may be available in a specific implementation, which is not limited by the embodiment of the present application.

S604: and calculating check data according to the N data blocks in the target copy, and writing the check data into the M check blocks.

Specifically, the M parity chunks are stored in the same stripe as the R copies of the data chunks. R is a positive integer greater than or equal to 2, and N and M are positive integers. Exemplarily, the present embodiment may include 3 copies, and the writing is completed after the backup data is written into the 3 copies respectively. And selecting one copy from the 3 copies as a target copy to calculate check data. The storage space (check block) for the check data is a pre-allocated storage space. After the check data is calculated, the check data is directly written into the check block.

Exemplarily, reference is again made to fig. 5. After the target replica is selected as the 3 rd replica, the check data is generated using the original data in

blocks

8, 9, 10 and 11 of the 3 rd replica, and the check data is written into blocks 12 and 13.

S605: and deleting the data in the other copies except the target copy in the strip.

Specifically, S605 is identical to S403, and is not described herein again.

Fig. 7 exemplarily shows a schematic structural diagram of a data backup device according to an embodiment of the present application. As shown in fig. 7, the data backup apparatus 70 may include at least: a backup module 710, a check module 720, and a delete module 730. Wherein:

the backup module 710 is used for writing the data to be backed up into the R copies; wherein each copy comprises N data blocks;

the checking module 720 is configured to calculate checking data according to the N data blocks in the target copy after the data to be backed up is written, and write the checking data into the M checking blocks; the M check blocks and the data blocks of the R copies are stored in the same stripe; r is a positive integer greater than or equal to 2, and N and M are positive integers;

a deleting module 730, configured to delete data in other copies in the stripe except for the target copy.

In one possible embodiment, the data backup apparatus 70 further includes: a setting module, configured to set a capacity size of each data block before the backup module 710 writes the data to be backed up into the R copies;

the backup module 710 is specifically configured to write the data to be backed up into the R copies according to the capacity size of the data block.

In a possible embodiment, the backup module 710 is specifically configured to write the data to be backed up into a first data block of each of the R copies according to the capacity size of the data block;

and when the first data block meets the sealing condition, writing the remaining data to be backed up into the next data block of each copy until all the data to be backed up are written.

In a possible embodiment, the sequestration condition is that the remaining capacity is below a preset threshold.

In a possible embodiment, the R copies are stored in different servers, and/or different storage spaces of the same server.

In one possible embodiment, the verification module 720 includes: a determining unit and a verifying unit. Wherein:

the determining unit is specifically used for determining a target copy according to a preset rule after the data to be backed up is written;

and the verification unit is specifically used for calculating verification data according to the N data blocks in the target copy and writing the verification data into the M verification blocks.

In a possible embodiment, the determining unit is specifically configured to select, as the target copy, the copy in which the data is successfully written according to a default copy arrangement order.

It should be noted that, when the data backup apparatus provided in the foregoing embodiment executes the data backup method, only the division of the functional modules is illustrated, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the data backup device and the data backup method provided by the above embodiments belong to the same concept, and details of implementation processes thereof are referred to in the method embodiments and are not described herein again.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

Please refer to fig. 8, which is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 8, the electronic device 80 may include: at least one processor 801, at least one network interface 804, a user interface 803, a memory 805, at least one communication bus 802.

Wherein a communication bus 802 is used to enable connective communication between these components.

The user interface 803 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 803 may also include a standard wired interface and a wireless interface.

The network interface 804 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface).

Processor 801 may include one or more processing cores, among other things. The processor 801 interfaces with various components throughout the electronic device 80 using various interfaces and lines to perform various functions of the electronic device 80 and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 805 and invoking data stored in the memory 805. Alternatively, the processor 801 may be implemented in at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 801 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content required to be displayed by the display screen; the modem is used to handle wireless communications. It is to be understood that the modem may not be integrated into the processor 801, but may be implemented by a single chip.

The Memory 805 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable medium. The memory 805 may be used to store instructions, programs, code sets, or instruction sets. The memory 805 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like; the storage data area may store data and the like referred to in the above respective method embodiments. The memory 805 may optionally be at least one memory device located remotely from the processor 801 as previously described. As shown in fig. 8, the memory 805, which is a type of computer storage medium, may include an operating system, a network communication module, a user interface module, and a data backup application.

In the electronic device 80 shown in fig. 8, the user interface 803 is mainly used as an interface for providing input for a user, and acquiring data input by the user; and the processor 801 may be configured to invoke the data backup application stored in the memory 805 and specifically perform the following operations:

deleting data in copies of the strip other than the target copy.

In a possible embodiment, before the processor 801 writes the data to be backed up into the R copies, the processor 801 is further configured to: setting the capacity of each data block;

when the processor 801 writes the data to be backed up into the R copies, the following steps are specifically executed: and writing the data to be backed up into R copies according to the capacity of the data block.

In a possible embodiment, when the processor 801 writes data to be backed up into R copies according to the capacity size of the data block, specifically perform:

writing the data to be backed up into a first data block of each of the R copies according to the capacity of the data block;

In a possible embodiment, after the data to be backed up is written into the target copy, the processor 801 calculates check data according to N data blocks in the target copy, and when the check data is written into M check blocks, specifically executes:

after the data to be backed up is written, determining a target copy according to a preset rule;

and calculating check data according to the N data blocks in the target copy, and writing the check data into the M check blocks.

In a possible embodiment, when the processor 801 determines the target copy according to a preset rule, the following specific steps are performed: and selecting the copy with successfully written data as a target copy according to a default copy arrangement sequence.

Embodiments of the present application also provide a computer-readable storage medium, which stores instructions that, when executed on a computer or a processor, cause the computer or the processor to perform one or more of the steps in the embodiments shown in fig. 4 to 5. The above-mentioned components of the data backup device may be stored in the computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)), or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., Digital Versatile Disk (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. And the aforementioned storage medium includes: various media capable of storing program codes, such as a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk. The technical features in the present examples and embodiments may be arbitrarily combined without conflict.

The above-described embodiments are merely preferred embodiments of the present application, and are not intended to limit the scope of the present application, and various modifications and improvements made to the technical solutions of the present application by those skilled in the art without departing from the design spirit of the present application should fall within the protection scope defined by the claims of the present application.

Claims

1. A method for data backup, comprising:

deleting data in copies of the strip other than the target copy.

2. The method of claim 1, wherein prior to writing the data to be backed up into the R copies, the method further comprises: setting the capacity of each data block;

the writing of the data to be backed up into the R copies includes: and writing the data to be backed up into R copies according to the capacity of the data block.

3. The method of claim 2, wherein writing the data to be backed up into R copies according to the size of the capacity of the data block comprises:

4. The method of claim 3, wherein the sequestration condition is a remaining capacity below a preset threshold.

5. The method according to claim 1, wherein the R copies are stored in different servers and/or different storage spaces of the same server, respectively.

6. The method as claimed in claim 1, wherein after the writing of the data to be backed up is completed, calculating parity data according to N data blocks in the target copy, and writing the parity data into M parity blocks, includes:

7. The method of claim 6, wherein the determining the target copy according to the preset rule comprises: and selecting the copy with successfully written data as a target copy according to a default copy arrangement sequence.

8. A data backup apparatus, comprising:

9. A computer storage medium, characterized in that it stores a plurality of instructions adapted to be loaded by a processor and to perform the method steps according to any of claims 1-7.

10. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps according to any of claims 1-7.