CN110765095A

CN110765095A - Ceph-based RBD data storage method, system, server and storage medium

Info

Publication number: CN110765095A
Application number: CN201911030323.4A
Authority: CN
Inventors: 龚立义; 谢俊
Original assignee: Data Technology (shenzhen) Ltd By Share Ltd
Current assignee: Data Technology (shenzhen) Ltd By Share Ltd
Priority date: 2019-10-28
Filing date: 2019-10-28
Publication date: 2020-02-07

Abstract

The embodiment of the invention discloses a Ceph-based RBD data storage method, a system, a server and a storage medium. The method comprises the following steps: obtaining RBD block data; splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata; and storing the plurality of RBD sub-data into at least one storage pool according to a second sequence. The invention provides safety protection for the RBD equipment in the Ceph distributed cluster file system, thereby achieving the technical effects that when the distributed cluster file system fails, data cannot be lost and the data can be recovered in time.

Description

Ceph-based RBD data storage method, system, server and storage medium

Technical Field

The embodiment of the invention relates to a data backup and recovery technology, in particular to a RBD data storage method, a system, a server and a storage medium based on Ceph.

Background

With the development of cluster storage technology, the distribution of cluster file systems is particularly important to the scalability of clusters. Most cluster file systems are realized in a mode of a central node and a metadata server. The defect is that the extension of the cluster file system is limited by a central node and a metadata server. Based on this background, Ceph is a good choice. Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability. Specifically, "unified" means that Ceph can provide three functions of object storage, block storage, and file system storage simultaneously in one storage system so as to meet different application requirements. Block storage supports snapshots, which generally are based on a point in time marking and then, when needed, restore state to the marked point, a premise that underlying things are not corrupted, where snapshots are used to save state on the storage system, and where snapshots of data can be successfully restored if the storage system is good and, once the storage system is bad, the snapshots are simultaneously stale.

In the prior art, the technical problems that once a backup and recovery technology of RBD equipment in a Ceph fails, data cannot be timely recovered and the data is lost exist.

Disclosure of Invention

The invention provides a Ceph-based RBD data storage method, a Ceph-based RBD data storage system, a server and a storage medium, and aims to realize that data cannot be lost and can be recovered in time when a distributed cluster file system fails.

In a first aspect, an embodiment of the present invention provides a Ceph-based RBD data storage method, including:

obtaining RBD block data;

splitting RBD block data according to a first sequence to obtain a plurality of RBD subdata;

the plurality of RBD sub-data is stored to the at least one storage pool in a second order.

Further, splitting the RBD block data according to the first order to obtain a plurality of RBD sub-data includes:

judging whether RBD block data can be averagely split into a plurality of RBD subdata with preset sizes or not;

if yes, splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata;

if not, determining the number of the RBD subdata according to the following formula, and splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata:

m＝INT(a/b)+1

wherein m represents the number of the RBD subdata, a represents the size of the RBD block data, b represents the size of each RBD subdata, and the INT function represents the downward rounding of the quotient obtained from the (a/b).

and encrypting the RBD block data to ensure the safety of the RBD block data.

Further, storing the plurality of RBD sub-data into the at least one storage pool in a second order comprises:

reading a plurality of RBD subdata in the storage pool;

and splicing the plurality of RBD subdata according to a first sequence to obtain RBD backup block data.

Further, obtaining the RBD backup block data includes:

and decrypting the RBD backup block data.

Further, the encryption includes symmetric encryption and asymmetric encryption.

Further, algorithms for symmetric encryption include AES, DES and 3DES, and algorithms for asymmetric encryption include RSA, DSA and ECC.

In a second aspect, an embodiment of the present invention further provides a Ceph-based RBD data storage system, including:

the acquisition module is used for acquiring RBD block data;

the splitting module is used for splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata;

and the storage module is used for storing the plurality of RBD subdata into at least one storage pool according to a second sequence.

In a third aspect, an embodiment of the present invention further provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of any one of the above-mentioned Ceph-based RBD data storage methods when executing the computer program.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of any one of the above-mentioned Ceph-based RBD data storage methods.

The invention provides safety protection for the RBD equipment in the Ceph distributed cluster file system, solves the technical problem that once the Ceph system fails, data cannot be recovered in time and data is lost in the prior art, and achieves the technical effects of effectively protecting the data, preventing the data from being lost and recovering the data in time aiming at the condition that the distributed cluster file system fails.

Drawings

Fig. 1 is a flowchart of a Ceph-based RBD data storage method according to an embodiment of the present invention;

fig. 2 is a flowchart of a Ceph-based RBD data storage method according to a second embodiment of the present invention;

fig. 3 is a schematic diagram illustrating a splitting of RBD block data according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of a Ceph-based RBD data storage system according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of a server according to a fourth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are for purposes of illustration and not limitation. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

Furthermore, the terms "first," "second," and the like may be used herein to describe various orientations, actions, steps, elements, or the like, but the orientations, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, a first split unit may be referred to as a second split unit, and similarly, a second split unit may be referred to as a first split unit, without departing from the scope of the present application. Both the first splitting unit and the second splitting unit are splitting units, but they are not the same splitting unit. The terms "first", "second", etc. are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Example one

Fig. 1 is a flowchart of a Ceph-based RBD data storage method according to an embodiment of the present invention, which is applicable to backup and read data. As shown in fig. 1, the method for storing the Ceph-based RBD data includes:

step S110, RBD block data is obtained;

in particular, Ceph is a unified, distributed file system designed for excellent performance, reliability, and scalability. The Ceph storage system can simultaneously provide three functions of object storage, block storage and file system storage so as to meet different application requirements. The RBD is a block device in the Ceph, the function of a 4T block device is similar to that of a 4T SATA, and the mounted RBD can be used as a disk.

Step S120, splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata;

specifically, before the RBD block data is split, the RBD block data can be encrypted first, so that the safety of the RBD fast data is ensured.

And S130, storing the plurality of RBD subdata into at least one storage pool according to a second sequence.

Specifically, when storing the RBD sub-data, the plurality of RBD sub-data may all be stored in different storage areas of the same storage pool, or the plurality of RBD sub-data may be stored in different storage pools.

The first embodiment of the invention has the advantages that safety protection is provided for RBD equipment in the Ceph distributed cluster file system, the technical problem that data cannot be recovered in time and data is lost once the Ceph system fails in the prior art is solved, and the technical effects of effectively protecting data, preventing data from being lost and recovering data in time aiming at the condition that the distributed cluster file system fails are achieved.

Example two

The second embodiment is further optimized on the basis of the first embodiment. Fig. 2 is a flowchart of a Ceph-based RBD data storage method according to a second embodiment of the present invention. As shown in fig. 2, the method for storing Ceph-based RBD data according to the present embodiment includes:

step S210, RBD block data is obtained;

Step S220, encrypting the RBD block data to ensure the safety of the RBD block data;

specifically, the RBD block data is encrypted before being split, so that the integrity and the safety of the RBD block data can be effectively ensured.

Step S230, judging whether the RBD block data can be averagely split into a plurality of RBD subdata with preset sizes;

specifically, whether the RBD block data can be split into a plurality of RBD subdata with the same size or not is judged firstly, and the RBD block data and the preset RBD subdata are divided into two splitting schemes for processing, so that the RBD block data can be correctly split.

Step S241, if yes, splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata;

step S242, if not, determining the number of RBD sub-data according to the following formula, and splitting the RBD block data according to the first order to obtain a plurality of RBD sub-data:

m＝INT(a/b)+1

wherein m represents the number of the RBD subdata, a represents the size of the RBD block data, b represents the size of each RBD subdata, and the INT function represents the downward rounding of the quotient obtained from the (a/b);

specifically, in this embodiment, one or more RBD block data may be used. Fig. 3 is a schematic diagram illustrating splitting of RBD block data according to a second embodiment of the present invention. As shown in fig. 3, one RBD block data 1 is 10M in size, and when the size of the RBD sub-data 10 is 2M, that is, one RBD block data 1 can be split into 5 2M RBD sub-data 11 on average, and when the size of the RBD sub-data is 4M, that is, one RBD block data 1 can be split into only 2 4M RBD sub-data 11 plus one remaining 2M RBD sub-data 12. When 2 pieces of RBD block data 2 are obtained, each RBD block data 2 is 10M in size, when the size of the RBD sub-data 21 is 2M, that is, 2 pieces of RBD block data 2 can be split into 10 pieces of 2M-sized RBD sub-data 21 on average, when the size of the RBD sub-data is 4M, when the first RBD block data 2a is split, the first RBD block data 2a is split into 2 pieces of 4M-sized RBD sub-data 21 and 1 piece of 2M-sized RBD sub-data 22a, and then the 1 piece of 2M-sized RBD sub-data 22a split from the first RBD block data 2a is filled by splitting the 2-th RBD block data 2b, that is, the second RBD sub-data 2b is split into one piece of 2M-sized RBD sub-data 22b in advance and is integrated with the 2M-sized RBD sub-data 22a of the first RBD block data 2a as one piece of 4M-sized RBD sub-data, and then the second RBD sub-data 2b is split into 2M, by the method, after all RBD block data are split, only the size of the last RBD subdata can be different from the sizes of other RBD subdata, and the integrity and the continuity of data splitting are guaranteed.

Step S243, storing the plurality of RBD subdata into at least one storage pool according to a second sequence;

specifically, after the RBD block data is split to obtain a plurality of RBD subdata, the plurality of RBD subdata may all be stored in different storage areas of the same storage pool, or may be stored in different storage pools.

Step S250, reading a plurality of RBD subdata in the storage pool;

specifically, after the storage of the RBD block data is completed, when the system needs to call the RBD block data, a plurality of RBD subdata in the storage pool needs to be read first.

Step S260, splicing the plurality of RBD subdata according to a first sequence to obtain RBD backup block data;

specifically, in step S241, the RBD block data is split according to the first order, and then in step S260, the RBD sub-data is also spliced according to the split order, so as to obtain RBD backup block data.

And step S270, decrypting the RBD backup block data.

Specifically, the RBD block data is encrypted in step S220, so that the RBD backup block data needs to be decrypted after being obtained.

In the present embodiment, the encryption includes symmetric encryption and asymmetric encryption.

Specifically, encryption techniques are generally classified into two categories, namely "symmetric" and "asymmetric," and symmetric encryption is encryption and decryption using the same key. Both parties of information receiving need to know the key and the encryption and decryption algorithm in advance, and the key is the same, then the data is encrypted and decrypted. Asymmetric encryption is that encryption and decryption use different keys, and usually there are two keys, called "public key" and "private key", which must be used in pair, otherwise the encrypted file cannot be opened. In the embodiment, a symmetric encryption algorithm DES is adopted, the data encryption standard of the algorithm is high in speed and suitable for occasions of encrypting a large amount of data, the algorithm is suitable for encrypting information such as sensitive data, and the like, and the encryption algorithm is changed through plug-in units so as to meet the requirements of different users.

In the present embodiment, the algorithms for symmetric encryption include AES, DES, and 3DES, and the algorithms for asymmetric encryption include RSA, DSA, and ECC.

Specifically, Advanced Encryption Standard (AES), also known as Rijndael Encryption method in cryptography, is a block Encryption Standard adopted by the federal government in the united states. The Data Encryption Algorithm (DEA) is a symmetric encryption Algorithm, most likely the most widely used key system, and in particular in securing financial data, the DEA originally developed is embedded in hardware. Generally, a DEA is used for an Automated Teller Machine (ATM). Triple Data Encryption Algorithm 3DES (or called TripleDES) is a generic term for (TDEA, Triple Data Encryption Algorithm) block ciphers. It is equivalent to applying the DES encryption algorithm three times per block. The RSA public key encryption algorithm was proposed in 1977 together by ronard listeriost (RonRivest), addi samor (Adi Shamir) and lunard Adleman (Leonard Adleman). RSA is a composition of their first three surnames in a spelled-together letter. The DSA algorithm (DSA) is a variant of the Schnorr and ElGamal signature algorithms, and is referred to by NIST in the United states as DSfS (Digital signature Standard). Error Correction Code (ECC) is a technology capable of implementing Error correction, and an ECC memory is a memory to which the technology is applied, and is generally applied to servers and graphic workstations, which makes the whole computer system more safe and stable during operation.

The second embodiment of the invention has the advantages that the RBD equipment in the Ceph distributed cluster file system is provided with safety protection, and different splitting methods are adopted according to the sizes of the RBD block data and the RBD subdata, so that the technical problem that data cannot be recovered in time and data is lost once the Ceph system fails in the prior art is solved, and the technical effects of effectively protecting the data, preventing the data from being lost and recovering the data in time aiming at the condition that the distributed cluster file system fails are achieved.

EXAMPLE III

Fig. 4 is a schematic structural diagram of a Ceph-based RBD data storage system according to a third embodiment of the present invention. As shown in fig. 4, the Ceph-based RBD data storage system 300 includes:

an obtaining module 310, configured to obtain RBD block data;

the splitting module 320 is configured to split the RBD block data according to a first order to obtain a plurality of RBD subdata;

the storage module 330 is configured to store the plurality of RBD sub-data into at least one storage pool according to a second order.

In this embodiment, the splitting module 320 includes: :

the first judgment unit is used for judging whether the RBD block data can be averagely split into a plurality of RBD subdata with preset sizes;

the first splitting unit is used for splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata if the RBD subdata is in the first sequence;

the second splitting unit is used for determining the number of the RBD subdata according to the following formula if the RBD subdata is not determined, and splitting the RBD block data according to the first sequence to obtain a plurality of RBD subdata;

m＝INT(a/b)+1

In this embodiment, the Ceph-based RBD data storage system 300 further includes:

and the encryption module 340 is configured to encrypt the RBD block data to ensure security of the RBD block data.

a reading module 350, configured to read a plurality of RBD subdata in the storage pool;

the splicing module 360 is configured to splice the plurality of RBD sub-data according to a first order to obtain RBD backup block data.

and a decryption module 370, configured to decrypt the RBD backup block data.

The Ceph-based RBD data storage system provided by the embodiment of the invention can execute the Ceph-based RBD data storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

Fig. 5 is a schematic structural diagram of a server according to a fourth embodiment of the present invention, as shown in fig. 5, the server includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of the processors 410 in the server may be one or more, and one processor 410 is taken as an example in fig. 4; the processor 410, the memory 420, the input device 430 and the output device 440 in the server may be connected by a bus or other means, and the bus connection is exemplified in fig. 4.

The memory 410, which is a computer-readable storage medium, may be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the Ceph-based RBD data storage system in the embodiments of the present invention (e.g., an acquisition module, a splitting module, a storage module, an encryption module, a reading module, a splicing module, and a decryption module in the Ceph-based RBD data storage system). The processor 410 executes various functional applications of the server and data processing by executing software programs, instructions, and modules stored in the memory 420, that is, implements the above-described Ceph-based RBD data storage method.

Namely:

obtaining RBD block data;

The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 420 may further include memory located remotely from processor 410, which may be connected to a server over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 430 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the server. The output device 440 may include a display device such as a display screen.

EXAMPLE five

An embodiment of the present invention further provides a storage medium containing computer-executable instructions, where the computer-executable instructions are executed by a computer processor to perform a Ceph-based RBD data storage method, where the method includes:

obtaining RBD block data;

Of course, the storage medium provided by the embodiments of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the above method operations, and may also perform related operations in the Ceph-based RBD data storage method provided by any embodiment of the present invention.

From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.

It should be noted that, in the embodiment of the above Ceph-based RBD data storage system, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments illustrated herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A Ceph-based RBD data storage method is characterized by comprising the following steps:

obtaining RBD block data;

splitting the RBD block data according to a first sequence to obtain a plurality of RBD subdata;

and storing the plurality of RBD sub-data into at least one storage pool according to a second sequence.

2. A Ceph-based RBD data storage method according to claim 1, wherein the splitting the RBD block data in a first order to obtain a plurality of RBD sub-data comprises:

judging whether the RBD block data can be averagely split into a plurality of RBD subdata with preset sizes or not;

if not, determining the number of the RBD subdata according to the following formula, and splitting the RBD block data according to a first sequence to obtain the plurality of RBD subdata:

m＝INT(a/b)+1

3. A Ceph-based RBD data storage method according to claim 1, wherein the splitting the RBD block data in the first order to obtain the plurality of RBD sub-data comprises:

and encrypting the RBD block data to ensure the safety of the RBD block data.

4. A Ceph-based RBD data storage method according to claim 1, wherein said storing the plurality of RBD sub-data in the second order to the at least one storage pool comprises:

reading the plurality of RBD subdata in the storage pool;

and splicing the plurality of RBD subdata according to the first sequence to obtain RBD backup block data.

5. A Ceph-based RBD data storage method according to claim 1, wherein obtaining RBD backup block data comprises:

and decrypting the RBD backup block data.

6. A Ceph-based RBD data storage method according to claim 3, wherein the encryption includes symmetric encryption and asymmetric encryption.

7. A Ceph-based RBD data storage method according to claim 6, characterized in that said symmetric encryption algorithms comprise AES, DES and 3DES, and said asymmetric encryption algorithms comprise RSA, DSA and ECC.

8. A Ceph-based RBD data storage system, comprising:

the acquisition module is used for acquiring RBD block data;

9. A server comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the computer program implements the steps of the Ceph-based RBD data storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, the computer program, when being executed by the processor, implementing the steps of the Ceph-based RBD data storage method according to any one of claims 1 to 7.