CN117633838A - File data encryption method and system for distributed storage - Google Patents

Abstract

The invention discloses a file data encryption method and system for distributed storage, belongs to the technical field of computer testing, and aims to solve the technical problem of improving the security of storage data in the distributed storage. For application to a distributed storage cluster comprising a data center and a plurality of data nodes, the method comprises the steps of: configuring an API interface; encrypting a file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension coding on the file blocks to obtain an extension file, performing data signing on the extension file to obtain a signature file, and storing the signature file in different data nodes; and acquiring corresponding signature files from each data center through the third party client, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain original files.

Description

File data encryption method and system for distributed storage

Technical Field

The invention relates to the technical field of computer testing, in particular to a file data encryption method and system for distributed storage.

Background

With the development of cloud computing and AI industry, distributed storage is gradually the mainstream choice of data center by virtue of the characteristics of low cost, high storage capacity and easy expansion, the distributed storage builds a distributed file system based on a high-speed interconnected storage server as hardware, and a plurality of storage nodes are combined into a storage pool by means of virtualization technology, so that unified storage service is provided for the outside, and the distributed storage has the advantages of convenience and simplicity in management, low technical difficulty, cost saving, convenience in expansion and the like. However, due to the characteristic of discrete distribution of the data structure of the distributed file system, the distributed storage in a single storage node is difficult to effectively encrypt the data, and potential safety hazards exist. Although the data center security architecture can prevent a certain degree of external attack through access control, it cannot prevent data from being leaked from the inside; secondly, data of a user are often stored in a cloud subsystem, so that the data are concentrated in a certain data center, the problem of single-point faults caused by network, power supply and the like exists, and once the cloud subsystem loses the data due to faults, the data cannot be recovered.

How to improve the security of data stored in distributed storage is a technical problem to be solved.

Disclosure of Invention

The technical task of the invention is to provide a file data encryption method and a file data encryption system for distributed storage, which aim at the defects, so as to solve the technical problem of how to improve the security of stored data in the distributed storage.

In a first aspect, the present invention provides a file data encryption method for distributed storage, applied to a distributed storage cluster including a data center and a plurality of data nodes, each cluster node is deployed with a distributed storage system, and a file in the distributed storage system is stored in a container in the form of an object, the method includes the following steps:

configuration API interface: defining an API interface specification and constructing an API interface, wherein the API interface is used for creating a container, reading and writing files and managing the files;

writing a file: acquiring a file to be written by a user through a read-write interface provided by a central node, encrypting the file to be written to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signature on the extension file to obtain a signature file, storing the signature file into different data nodes, and managing the file, corresponding file blocks, the extension file and the signature file through metadata in the central node for the files stored in each different data node;

reading a file: and for the file to be read, acquiring corresponding signature files from each data center through a third party client, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain the original file.

Preferably, the API interface manages the file including file querying, file encryption, file splitting, and file signing.

Preferably, the file writing includes the steps of:

encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

for each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

for each extension file, signing the extension file to obtain a signature file;

correspondingly, the data reading comprises the following steps:

acquiring corresponding signature files from each data center through a third party client;

checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

combining the effective extension files to obtain a ciphertext file;

cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

and decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

Preferably, when writing a file, performing extension coding on a file block to obtain an extension file, performing exclusive-or calculation on the extension file to obtain a check value, and storing the check value in a data center;

when the file is read, when the signature file is checked through the digital signature, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node, the central node carries out reverse pushing on the relevant check value based on the invalidation notification to obtain the corresponding extension file, the extension file is signed again to obtain a new signature file, and the new signature file is sent to the corresponding data node and covers the previous signature file.

In a second aspect, the present invention provides a file data encryption system for distributed storage, applied to a distributed storage cluster including a data center and a plurality of data nodes, where each cluster node is deployed with a distributed storage system, where a file in the distributed storage system is stored in a container in the form of an object, and used for encryption by a file data encryption method for distributed storage according to any one of the first aspects, where the encryption system includes a configuration module, a writing module, and a reading module, and each cluster node is configured with the configuration module, the writing module, and the reading module;

the configuration module is used for defining an API interface specification and constructing an API interface for the distributed file system, and the API interface is used for creating a container, reading and writing files and managing the files;

when writing a file, the central node acquires the file to be written by a user through a read-write interface provided by the central node, and for the file to be written, a write module in the central node is matched with an API interface to execute the following steps: encrypting a file to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signing on the extension file to obtain a signature file, storing the signature file in different data nodes, and managing the file, the corresponding file blocks, the extension file and the signature file through metadata at a central node for files stored in each different data node;

when the file is read, the file reading module in the third party client is used for executing the following steps of: and acquiring corresponding signature files from each data center, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain original files.

Preferably, the API interface manages the file including file querying, file encryption, file splitting, and file signing.

Preferably, the file writing module in the central node is matched with the API interface to execute the following operations:

encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

for each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

for each extension file, signing the extension file to obtain a signature file;

correspondingly, the data reading module is matched with the API interface to execute the following operations:

acquiring corresponding signature files from each data center through a third party client;

checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

combining the effective extension files to obtain a ciphertext file;

cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

and decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

Preferably, when writing a file, the file writing module in the central node is matched with the API interface to execute the following steps: after the file block is subjected to expansion coding to obtain an expansion file, the file writing module is matched with the API interface to perform exclusive-or calculation on the expansion file to obtain a check value, and the check value is stored in a data center;

when the file is read, the data reading module in the third party client is matched with the API interface to execute the following steps: when the signature file is checked through the digital signature, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node; correspondingly, based on the invalidation notification, the data reading module in the central node is configured to perform the following: and carrying out reverse pushing on the related check values to obtain corresponding extension files, carrying out signature on the extension files again to obtain new signature files, sending the new signature files to corresponding data nodes, and enabling a data reading module in the data nodes to receive the new signature files and cover the previous signature files.

The file data encryption method and system for distributed storage has the following advantages:

1. the method comprises the steps that a file to be written in is obtained through a central node, a ciphertext is formed by encrypting the file, the ciphertext is divided into a plurality of file blocks, each file block is subjected to expansion coding to obtain an expansion file, the expansion file is signed and then sent to a data node, and when the file is read, the received signature file is required to be checked, the effective expansion file is combined into the ciphertext and decrypted to obtain an original file, and the confidentiality, the reliability and the integrity of data in the file are ensured through encryption, expansion coding and signature;

2. when writing a file, performing exclusive or operation on the extended file to obtain a check value, and when reading the file, if the digital signature verification is not passed on the received signature file, reversely pushing the corresponding check value at the central node to obtain the extended file, signing the extended file and then sending the extended file to the data node, thereby realizing data recovery.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

The invention is further described below with reference to the accompanying drawings.

Fig. 1 is a flow chart of a file data encryption method for distributed storage according to embodiment 1.

Detailed Description

The invention will be further described with reference to the accompanying drawings and specific examples, so that those skilled in the art can better understand the invention and implement it, but the examples are not meant to limit the invention, and the technical features of the embodiments of the invention and the examples can be combined with each other without conflict.

The embodiment of the invention provides a file data encryption method and a file data encryption system for distributed storage, which are used for solving the technical problem of how to improve the security of stored data in the distributed storage.

Example 1:

the invention discloses a file data encryption method oriented to distributed storage, which is applied to a distributed storage cluster comprising a data center and a plurality of data nodes, wherein each cluster node is provided with a distributed storage system, and files in the distributed storage system are stored in a container in the form of objects. The method of the embodiment comprises three steps of configuration of an API interface, file writing and file reading.

Step S100 configures the API interface: define API interface specifications and build API interfaces for creating containers, reading and writing files, and managing operations.

The API interface manages the file, including file inquiry, file encryption, file segmentation and file signature.

The data read-write of the distributed storage system must use the API interfaces of different storage service providers, and the existing distributed storage uses a multi-purpose RESTFUL style interface. The scheme virtualizes the distributed storage systems of different storage manufacturers into a unified storage interface, and can perform operations such as reading, writing, searching and the like on the different distributed storage systems through the interface. All files on the distributed storage system are one object (object). The objects may be placed in different containers (containers) that are similar to the root directory of a file system. With reference to the distributed storage system Ceph architecture of the industry, we design API interfaces to include creation of containers, object querying, object manipulation, large file splitting, object writing, etc. With these definable interfaces and related parameters, data can be stored in different data nodes.

Step S200, writing a file: the method comprises the steps of obtaining a file to be written by a user through a read-write interface provided by a central node, encrypting the file to be written to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signature on the extension file to obtain a signature file, storing the signature file into different data nodes, and managing the file, corresponding file blocks, the extension file and the signature file through metadata in the central node for files stored in different data nodes.

As a specific implementation, the file writing includes the following steps:

(1) Encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

(2) Dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

(3) For each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

(4) And signing the extension files for each extension file to obtain signature files.

When the file is written, the expansion coding is carried out on the file blocks to obtain an expansion file, then the exclusive OR calculation is carried out on the expansion file to obtain a check value, and the check value is stored in the data center.

The distributed storage consists of a central node and a plurality of data nodes. The central node shields the user from the differentiated interfaces of the plurality of data nodes, and the user can use a storage pool formed by a plurality of data nodes at a remote place like a local disk.

When the user is about to write the file, the file is firstly encrypted through a symmetrical algorithm, the encryption algorithm is AES-256, and the length of the ciphertext file is recorded. The ciphertext file is divided into x file blocks with the same size through a segmentation algorithm, and files with the sizes smaller than one file block are filled to the size of one file block by 0.

And then carrying out cauchy matrix extension coding on the encrypted x file blocks to obtain x extension files, and carrying out data signature on the x extension files to obtain x signature files. And finally, storing the signed encrypted file on different data nodes through a high-speed internet.

Step S300, file reading: and for the file to be read, acquiring corresponding signature files from each data center through a third party client, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain the original file.

As a specific implementation, the data reading includes the following steps:

(1) Acquiring corresponding signature files from each data center through a third party client;

(2) Checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

(3) Combining the effective extension files to obtain a ciphertext file;

(4) Cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

(5) And decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

In this embodiment, when a file needs to be read, the client reads a certain number of encrypted files from a plurality of data nodes, and checks the encrypted files with the digital signature of the encrypted files when one of the encrypted files is read, and the verification indicates that the encrypted file is damaged or tampered by the verification failure through indicating that the encrypted file is a valid encrypted file.

And combining all the effective extension files, decoding to obtain a ciphertext file filled with 0, cutting according to the length of the file recorded before to obtain an original ciphertext file, and finally decrypting the ciphertext by using a secret key to obtain the original file.

When the signature file is checked through the digital signature during file reading, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node, the central node carries out reverse pushing on the relevant check value based on the invalidation notification to obtain the corresponding extension file, signs the extension file again to obtain a new signature file, and sends the new signature file to the corresponding data node and covers the previous signature file.

In this embodiment, the missing extension files in the distributed file system are recovered. The repair is to ensure the redundancy of the data and prevent the original file from being unreadable due to the fact that the invalid redundant file exceeds the coding redundancy.

Example 2:

the invention discloses a file data encryption system oriented to distributed storage, which is applied to a distributed storage cluster comprising a data center and a plurality of data nodes, wherein each cluster node is provided with the distributed storage system, files in the distributed storage system are stored in a container in the form of objects, the files are encrypted by the method disclosed in the embodiment 1, and the encryption system comprises a configuration module, a writing module and a reading module, and each cluster node is provided with the configuration module, the writing module and the reading module.

The configuration module is used for defining an API interface specification and constructing an API interface for the distributed file system, wherein the API interface is used for creating a container, reading and writing files and managing the files.

The data read-write of the distributed storage system must use the API interfaces of different storage service providers, and the existing distributed storage uses a multi-purpose RESTFUL style interface. The scheme virtualizes the distributed storage systems of different storage manufacturers into a unified storage interface, and can perform operations such as reading, writing, searching and the like on the different distributed storage systems through the interface. All files on the distributed storage system are one object (object). The objects may be placed in different containers (containers) that are similar to the root directory of a file system. With reference to the distributed storage system Ceph architecture of the industry, we design API interfaces to include creation of containers, object querying, object manipulation, large file splitting, object writing, etc. With these definable interfaces and related parameters, data can be stored in different data nodes.

The API interface manages files including file querying, file encryption, file splitting, and file signing.

When writing a file, the central node acquires the file to be written by a user through a read-write interface provided by the central node, and for the file to be written, a write module in the central node is matched with an API interface to execute the following steps: encrypting a file to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signing on the extension file to obtain a signature file, storing the signature file in different data nodes, and managing the file, the corresponding file blocks, the extension file and the signature file through metadata at a central node for files stored in different data nodes.

In this embodiment, the file writing module in the central node cooperates with the API interface to perform the following operations:

(1) Encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

(2) Dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

(3) For each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

(4) And signing the extension files for each extension file to obtain signature files.

As an improvement, when writing a file, the file writing module in the central node is matched with the API interface to execute the following steps: after the file block is subjected to expansion coding to obtain an expansion file, the file writing module is matched with the API interface to perform exclusive-or calculation on the expansion file to obtain a check value, and the check value is stored in the data center.

When the file is read, the file reading module in the third party client is used for executing the following steps of: and acquiring corresponding signature files from each data center, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain original files.

In this embodiment, the data reading module cooperates with the API interface to perform the following operations:

(1) Acquiring corresponding signature files from each data center through a third party client;

(2) Checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

(3) Combining the effective extension files to obtain a ciphertext file;

(4) Cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

(5) And decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

As an improvement, when the file is read, the data reading module in the third party client is matched with the API interface to execute the following steps: when the signature file is checked through the digital signature, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node; correspondingly, based on the invalidation notification, the data reading module in the central node is configured to perform the following: and carrying out reverse pushing on the related check values to obtain corresponding extension files, carrying out signature on the extension files again to obtain new signature files, sending the new signature files to corresponding data nodes, and enabling a data reading module in the data nodes to receive the new signature files and cover the previous signature files.

While the invention has been illustrated and described in detail in the drawings and in the preferred embodiments, the invention is not limited to the disclosed embodiments, and it will be appreciated by those skilled in the art that the code audits of the various embodiments described above may be combined to produce further embodiments of the invention, which are also within the scope of the invention.

Claims (8)

1. A distributed storage oriented file data encryption method, which is characterized by being applied to a distributed storage cluster comprising a data center and a plurality of data nodes, wherein each cluster node is provided with a distributed storage system, and files in the distributed storage system are stored in a container in the form of objects, and the method comprises the following steps:

configuration API interface: defining an API interface specification and constructing an API interface, wherein the API interface is used for creating a container, reading and writing files and managing the files;

writing a file: acquiring a file to be written by a user through a read-write interface provided by a central node, encrypting the file to be written to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signature on the extension file to obtain a signature file, storing the signature file into different data nodes, and managing the file, corresponding file blocks, the extension file and the signature file through metadata in the central node for the files stored in each different data node;

reading a file: and for the file to be read, acquiring corresponding signature files from each data center through a third party client, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain the original file.

2. The distributed storage oriented file data encryption method of claim 1, wherein the API interface manages the file including file querying, file encryption, file splitting, and file signing.

3. The distributed storage oriented file data encryption method of claim 1, wherein the file writing comprises the steps of:

encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

for each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

for each extension file, signing the extension file to obtain a signature file;

correspondingly, the data reading comprises the following steps:

acquiring corresponding signature files from each data center through a third party client;

checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

combining the effective extension files to obtain a ciphertext file;

cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

and decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

4. A file data encryption method for distributed storage according to any one of claims 1-3, wherein when writing a file, after performing extension encoding on a file block to obtain an extension file, performing exclusive-or calculation on the extension file to obtain a check value, and storing the check value in a data center;

when the file is read, when the signature file is checked through the digital signature, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node, the central node carries out reverse pushing on the relevant check value based on the invalidation notification to obtain the corresponding extension file, the extension file is signed again to obtain a new signature file, and the new signature file is sent to the corresponding data node and covers the previous signature file.

5. A file data encryption system facing distributed storage, which is characterized by being applied to a distributed storage cluster comprising a data center and a plurality of data nodes, wherein each cluster node is provided with the distributed storage system, files in the distributed storage system are stored in a container in the form of objects, the distributed storage-oriented file data encryption system is used for encrypting by the file data encryption method facing distributed storage according to any one of claims 1-4, the encryption system comprises a configuration module, a writing module and a reading module, and each cluster node is provided with the configuration module, the writing module and the reading module;

the configuration module is used for defining an API interface specification and constructing an API interface for the distributed file system, and the API interface is used for creating a container, reading and writing files and managing the files;

when writing a file, the central node acquires the file to be written by a user through a read-write interface provided by the central node, and for the file to be written, a write module in the central node is matched with an API interface to execute the following steps: encrypting a file to obtain a ciphertext file, dividing the ciphertext file into a plurality of file blocks with the same size, performing extension encoding on the file blocks to obtain an extension file, performing data signing on the extension file to obtain a signature file, storing the signature file in different data nodes, and managing the file, the corresponding file blocks, the extension file and the signature file through metadata at a central node for files stored in each different data node;

when the file is read, the file reading module in the third party client is used for executing the following steps of: and acquiring corresponding signature files from each data center, checking the signature files through digital signatures to obtain effective extension files, merging the effective extension files to obtain ciphertext files, and decrypting the ciphertext files to obtain original files.

6. The distributed storage oriented file data encryption system of claim 5, wherein the API interface manages files including file querying, file encryption, file splitting, and file signing.

7. The distributed storage oriented file data encryption system of claim 5 wherein the file write module in the central node cooperates with the API interface to perform the following operations:

encrypting a file to be written by a symmetric encryption algorithm to obtain a ciphertext file, and recording the ciphertext length;

dividing the ciphertext file into a plurality of file blocks with the same size, and filling the file blocks with insufficient size;

for each file block, performing extension coding on the file block through a cauchy matrix to obtain an extension file;

for each extension file, signing the extension file to obtain a signature file;

correspondingly, the data reading module is matched with the API interface to execute the following operations:

acquiring corresponding signature files from each data center through a third party client;

checking the signature file through the digital signature, and if the signature file passes the checking, the corresponding extension file is an effective extension file;

combining the effective extension files to obtain a ciphertext file;

cutting the ciphertext file according to the recorded ciphertext length to obtain an original ciphertext file;

and decrypting the obtained ciphertext file through a symmetric encryption algorithm to obtain an original file.

8. The distributed storage oriented file data encryption system of claim 5 wherein the file write module in the central node cooperates with the API interface to perform the following: after the file block is subjected to expansion coding to obtain an expansion file, the file writing module is matched with the API interface to perform exclusive-or calculation on the expansion file to obtain a check value, and the check value is stored in a data center;

when the file is read, the data reading module in the third party client is matched with the API interface to execute the following steps: when the signature file is checked through the digital signature, if the corresponding extension file is invalid, an invalidation notification is sent to the central node through the data node; correspondingly, based on the invalidation notification, the data reading module in the central node is configured to perform the following: and carrying out reverse pushing on the related check values to obtain corresponding extension files, carrying out signature on the extension files again to obtain new signature files, sending the new signature files to corresponding data nodes, and enabling a data reading module in the data nodes to receive the new signature files and cover the previous signature files.