CN108400970B - Similar data message locking, encrypting and de-duplicating method in cloud environment and cloud storage system - Google Patents

Abstract

The invention belongs to the technical field of cloud storage, and discloses a method for locking, encrypting and deduplicating similar data messages in a cloud environment. The encryption algorithm constitutes the technical route of the present invention. Compared with the existing data deduplication method, the implementation of this scheme can further improve the efficiency of the existing scheme for deduplication, can further improve the utilization rate of cloud server storage space, and can further reduce the computing overhead of users and cloud servers. and storage overhead. The invention can realize safe and efficient deduplication of similar ciphertext data; the invention also adopts the Hamming distance reduction and label cutting optimization method, and can improve the cloud server's ability to perform label query through the Hamming distance reduction and label cutting optimization method. The experimental results show that the present invention is efficient in terms of storage and communication costs.

Description

Similar data message locking, encryption and deduplication method in cloud environment, cloud storage system

technical field

The invention belongs to the technical field of cloud storage, and in particular relates to a method for locking, encrypting and deduplicating similar data messages in a cloud environment, and a cloud storage system.

Background technique

Today, huge amounts of data are generated and processed every day. Research by the International Data Corporation on the digital realm points out that by 2020, the data on the Internet will reach 40,000 exabytes, and the data will continue to double at a rate of doubling every two years. Cloud computing brings a paradigm shift in data storage solutions. It brings great convenience to personal and enterprise data management by providing reliable, scalable, on-demand cloud storage services at relatively low prices. The Cisco Global Cloud Index states that 83 percent of all data center traffic originates in the cloud, and by 2019, 80 percent of data center workloads will be processed on the cloud. According to current research, 20% to 30% of the data in main memory is redundant. Specifically, by implementing the deduplication technology of all files in the backup storage, it will save more than 50% of the storage space of the standard file system and more than 72% of the storage space of the backup file system. Therefore, data deduplication technology can effectively reduce the pressure of data storage, reduce network traffic by removing redundant data, and improve system service quality. A series of online/offline storage systems already provide data deduplication, such as IBM's InfoSphere QualityStage service and FirstLogic's SAP Data Services and other commercial data integration tools support duplicate data detection. There are many clustering based techniques, classification techniques, link analysis techniques or statistical techniques used to detect duplicate records. There are also plenty of software designed to detect and eliminate identical or similar duplicate data, such as Duplicate Cleaner, VisiPics and DupeGuru. However, since users will lose physical control of their data in the cloud storage system, the security of user data becomes the biggest appeal. Therefore, in order to protect the security of cloud users' sensitive data, users usually encrypt the data before outsourcing it. However, the goal of cryptography is to provide the semantic security of plaintext data, making ciphertext data indistinguishable from random data. Therefore, in a multi-user cloud storage system, how to achieve data deduplication under the premise of protecting data security has become a key and challenging issue. To solve this problem, convergent encryption schemes are proposed. In convergent encryption, by using the hash value of the file as the convergent key, the same data can always get the same key. By using the convergent key to encrypt and decrypt the data, this enables ciphertext deduplication. This encryption scheme is formally defined as message locking encryption. By obtaining the same key from the same data for data encryption, the cloud server can determine whether two ciphertext data are obtained from the same plaintext. Since then, a series of new message locking encryption schemes have tried to improve the security of the scheme or provide other new features. However, those schemes only consider the deduplication of the same data and cannot be applied to realize the deduplication of similar data. Many practical systems require deduplication or searching for similar data items such as errors, spelling errors, and content inconsistencies in cases such as data inspection, data cleaning, and data aggregation. Some similar data retrieval schemes and deduplication systems have been proposed, and many schemes and software are used to delete similar content of web pages, text documents, music, pictures, videos or binary data on local disks. However, those schemes and software mainly address deduplication of plaintext data rather than ciphertext data. Further, their scheme is suitable for personal use but not for multi-user scenarios in cloud environment. Therefore, the existing solutions are difficult to be directly applied to secure similar data deduplication in the cloud environment. Although there are currently schemes to support similar image deduplication in a privacy-preserving cloud environment, they assume that there is a user group and that encryption keys are shared among the group. However, it is difficult for users in a cloud environment to know other users who have the same data. Generally speaking, the challenge of deduplication of similar data in a cloud environment is that it is difficult for cloud users to communicate and negotiate a common encryption key with other users, and it is also difficult for cloud servers to determine whether two ciphertexts are encrypted by similar data. of.

To sum up, the problems existing in the prior art are :

(1) The key of the existing message locking encryption scheme is obtained by calculating the hash value of its plaintext, and the characteristic of the hash function is that even if the plaintext is different by 1 bit, the obtained hash value is completely different. Therefore, the ciphertext encrypted by the traditional message locking encryption scheme is no longer similar, and the cloud server cannot judge whether the plaintext of the two ciphertext data is similar, so the existing message locking encryption scheme is difficult to directly apply to the cloud environment Deduplication of similar data under safe.

(2) On the other hand, although there are some solutions that can realize group users negotiate keys and share them within the group, users can upload data anytime and anywhere in the cloud environment, and the cloud server cannot know that all the data has ownership before users upload data. Therefore, the scheme of group users negotiating keys together cannot be used to deduplicate similar data.

The difficulty and significance of solving the above technical problems :

(1) How to break through the limitations of the existing message locking encryption scheme, so that similar data remains similar after encryption, is a problem that needs to be solved by the similar data message locking encryption and deduplication method.

(2) By realizing the structure of the similar data message locking encryption scheme, it can be used to realize the similar data encryption and deduplication system, so that the cloud server can realize the ciphertext deduplication of similar data, which will further improve the efficiency of ciphertext deduplication, Save a lot of storage resources and management resources of cloud servers.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the present invention provides a method for locking, encrypting and deduplicating similar data messages in a cloud environment, and a cloud storage system.

The present invention is implemented as follows: a method for locking, encrypting and deduplicating similar data messages in a cloud environment, wherein the method for locking, encrypting and deduplicating similar data messages in a cloud environment uses a similarity preserving hash function (such as SimHash or PHash) to make similar Data can get similar labels. The key extraction method based on error correction code makes it possible to always obtain the same encryption key for data with similar plaintext. The secure symmetric encryption algorithm based on pseudo-random generator locks, encrypts and deduplicates similar data messages; If you want to upload data, first use the similarity-preserving hash algorithm to generate a plaintext deduplication label and send it to the cloud server. The cloud server determines whether there is similar data already stored on the cloud server. If the cloud server does not have similar data, Users are required to generate similar data keys and auxiliary information for similar key recovery, and send the encrypted ciphertext data and auxiliary information to the cloud server; The auxiliary information of the key is given to the user. The user encrypts the data with the recovered similar key, and uses the obtained ciphertext to verify the ownership of the similar data with the server. If the verification is passed, the cloud server allows the user to access the data. In addition, the present invention also improves the efficiency of label query by means of Hamming distance reduction and label cutting optimization.

Further, the method for locking, encrypting and deduplicating similar data messages in the cloud environment includes the following steps:

The client first uses the similarity-preserving hash algorithm to generate a clear-text deduplication label and send it to the cloud server, and the cloud server determines whether there is similar data already stored on the cloud server;

If the cloud server does not have similar data, the user is required to generate a similar data key and auxiliary information for similar key recovery, and send the encrypted ciphertext data and auxiliary information to the cloud server;

If the cloud server has similar data, it will return auxiliary information for recovering the similar key to the user. The user encrypts the data with the recovered similar key, and uses the obtained ciphertext to verify the ownership of the similar data with the server. After verification, the cloud server allows the user to access the data.

Further, the method of locking, encrypting and deduplicating similar data messages in the cloud environment uses the error correction code C of [n, k, 2t+1] _F , and the idea of the security sketch based on the Hamming distance is to use the error correction code C to correct the data w Perform error correction; input w, uniformly and randomly select the codeword c∈C, let s=SS(w)=wc be the transformation required from c to w; calculate Rec(w',s), through the formula c'=w' -s then decode c' to get c; get w by w=c+s.

Further, in the cloud environment, similar data messages are locked, encrypted, and deduplicated. The client applies a similarity-preserving hash algorithm to generate a plaintext deduplication label and a similar data key; using the similarity-preserving hash, similar plaintext data will be mapped to to similar labels and similar data keys with a certain length; similar data within a certain Hamming distance always get the same random encryption key, the first user chooses some auxiliary parameters and computes the random key for the plaintext w'; Auxiliary parameters will be stored on the cloud server; when a subsequent user has a tag tw of similar plaintext data _w (w≈w') and wants to perform a similar data deduplication operation, the cloud server will send the auxiliary parameter to the subsequent user, Subsequent users generate the key k _w by running the key regeneration algorithm; if the Hamming distance between the file w and the file w' is less than a specific value, (f _w , f _w' )<t, the key regeneration algorithm will The same random key _kw' = _kw is output.

Further, the similar message locking encryption scheme of the similar data message locking encryption and deduplication method in the cloud environment is composed of six polynomial time algorithms (FKG, KG, REP, ENC, DEC, TAG):

FKG(1 ^λ ,r ₂ ,w)→fk _w : It is a similar key generation algorithm based on similarity retention hash function, which is used to allow users to calculate the summary information of data; with security parameters λ, random numbers r ₂ ∈ { 0,1} ^λ and file w as input, output a similar summary fk _w of a file;

RKG(1 ^λ ,r ₃ ,fk _w )→{k _w ,P _w }: It is a key generation algorithm, which is used to allow users to calculate the encryption key and auxiliary parameters of the data; x is a public parameter, and the RKG algorithm uses The sketch algorithm SS{r ₃ ,w}→P _w of the secure sketch and the extraction algorithm Ext( _w ,x)→{Kw} in the fuzzy extractor generate auxiliary parameters P={x,s} and a random encryption key K _w , where r ₃ is a random parameter used to generate a random code C(r ₃ )→c algorithm C(·) is a code generation algorithm, and the code c is used for the SS algorithm in the security sketch;

REP(fk _w' ,P _w )→k _w : is a key regeneration algorithm, run by the user, by taking the auxiliary parameter P _w and the fuzzy digest fk _w' of the file as input, if and only if fk _w' and fk When _w is similar, output the private key k _w ; otherwise, output a random value;

其中G(k_w)→{0,1}^|w|是伪随机生成器，以k_w作为输入并输出长度为|w|的伪随机加密密钥G(k_w)；ENC(k _w ,w)→c _w : is an encryption algorithm, which is run by the user to calculate the encrypted data and obtain the corresponding ciphertext, taking the file _w and a private key kw as input, and returning the ciphertext

where G(k _w )→{0,1} ^|w| is a pseudo-random generator that takes k _w as input and outputs a pseudo-random encryption key G(k _w ) of length |w|;

DEC(k _w ,c _w )→w: is a decryption algorithm run by the user to calculate the plaintext of the input data; it takes the ciphertext c _w and a private key k _w as input, and returns the plaintext

TAG(1 ^λ ,r ₁ ,w)→t _w : is a tag generation algorithm, implemented by using a similarity-preserving hash function, run by the user to compute a digest of the input data. It takes as input a security parameter λ, a random number r ₁ and data w and returns a data label _tw .

Another object of the present invention is to provide a cloud storage system applying the method for locking, encrypting, and deduplicating similar data messages in the cloud environment.

To sum up, the advantages and positive effects of the present invention are as follows: a safe and efficient scheme for deduplicating similar data, called Fuzzy Message Locking Encryption Scheme (FuzzyMLE); using a similarity preserving hash function, based on error correction codes The method for proposing the key and the secure symmetric encryption algorithm based on the pseudo-random generator constitute the technical route of the present invention. In addition, the label query efficiency is improved through Hamming distance reduction and label cutting optimization. Finally, the efficiency of the present invention is analyzed, and the cost of the present invention is evaluated on a public database by building an actual system. Experimental results show that the present invention is efficient in terms of storage and communication overhead.

The present invention aims at safe and efficient cross-user data deduplication for similar data. If the cloud server has stored the data of user A, and the data of user B is similar to the data of user A, the cloud server can realize ciphertext deduplication without communicating with user A. A similar message-locking encryption scheme is formally defined and a similar message-locking encryption system is constructed. By improving and combining multiple technologies, the challenge of safe and efficient deduplication of similar duplicate data in cloud storage systems is overcome. First, compared with traditional cryptographic hash tags used in traditional message locking encryption schemes, similarity-preserving hash function is used to process similar data, and a similarity tag is generated for each data. Second, instead of the same label query, Hamming label query is improved and used to provide an efficient similar data query function. At the same time, a similar encryption key generation method based on error correction code is used to generate similar data encryption keys according to user data under the condition of similar data. Also, a secure XOR encryption scheme based on a pseudo-random generator is used instead of a conventional symmetric encryption algorithm (eg, AES) to implement the encryption operation. In addition, the present invention further improves the label query efficiency by introducing the Hamming distance reduction and label cutting optimization methods.

A cloud storage system consists of a remote cloud storage server (S) and a set of clients (Cs) who wish to store sensitive data on S. To keep the data safe, Cs want to encrypt their sensitive data before uploading it. In order to reduce the storage overhead and unnecessary communication overhead between S and Cs, S and Cs should implement secure deduplication of the uploaded ciphertext. Different from the existing secure duplicate data deduplication method that can only perform secure duplicate data deduplication on the same data, the present invention considers a more challenging situation: secure duplicate data deduplication of similar data. To improve communication efficiency, if Cs has stored some similar data in S's database, only the first user needs to upload the ciphertext of the data to S. In fact, similar to most existing accurate and secure deduplication methods, users do not need to communicate with each other directly in the system. They respectively communicate with S, which processes messages or forwards them when needed.

Description of drawings

FIG. 1 is a flowchart of a method for locking, encrypting, and deduplicating similar data messages in a cloud environment provided by an embodiment of the present invention.

FIG. 2 is a schematic diagram of a security sketch provided by an embodiment of the present invention.

FIG. 3 is a schematic diagram of a blur extractor provided by an embodiment of the present invention.

FIG. 4 is a schematic diagram of similar data locking and encryption provided by an embodiment of the present invention.

FIG. 5 is a schematic diagram of calculation time of SimHash provided by an embodiment of the present invention.

FIG. 6 is a schematic diagram of a PHash calculation time provided by an embodiment of the present invention.

7 is a schematic diagram of the time spent on deduplication of text data provided by an embodiment of the present invention

FIG. 8 is a schematic diagram of the time spent in deduplication of image data provided by an embodiment of the present invention.

FIG. 9 is a schematic diagram of a testing hardware environment provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

With the explosive growth of data, the efficiency of data storage has become the most important goal that cloud storage systems need to achieve. Most cloud storage providers use data deduplication to reduce data storage and management overhead. In recent years, in order to further protect the privacy of user data, many secure deduplication methods have been proposed. At the same time, many practical applications point out that eliminating similar (or erroneous) data can further reduce storage overhead for cloud storage providers and can improve data storage quality. However, there is still a lack of safe and efficient methods for deduplication of similar data in cloud storage environments.

As shown in FIG. 1 , the method for locking, encrypting and deduplicating similar data messages in a cloud environment provided by an embodiment of the present invention includes the following steps:

S101: The client first uses a similarity-preserving hash algorithm to generate a plaintext deduplication label and send it to the cloud server, and the cloud server determines whether there is similar data already stored on the cloud server.

S102: If the cloud server does not possess similar data, the user is required to generate a similar data key and auxiliary information for similar key recovery, and send the encrypted ciphertext data and auxiliary information to the cloud server.

S103: If the cloud server has similar data, return auxiliary information for recovering the similar key to the user, and the user encrypts the data with the recovered similar key, and uses the obtained ciphertext to verify the ownership of the similar data with the server, If verified, the cloud server allows the user to access the data.

The application principle of the present invention will be further described below with reference to the accompanying drawings.

1. Safety outline

The security sketch can reconstruct similar data and obtain the same data exactly with auxiliary information. Let M be the metric space of the distance function dis, Figure 2 depicts a schematic diagram of the security sketch. It is defined as follows:

A secure sketch with parameters (M, m, m', t, ) consists of two efficient random sketch algorithms and a recovery algorithm (SS, Rec).

The sketch algorithm SS: takes an element w∈M as input and outputs a string s∈{0,1} ^* .

The recovery algorithm Rec: takes the element w'∈M and the string s∈{0,1} ^* as the most input. When dis(w,w')≤t, then Rec(w',SS(w))=w, and when dis(w,w')≥t, the output of Rec is not guaranteed.

Hamming Distance Based Safe Sketch Algorithm: In order to obtain a safe sketch from the ^Fn Hamming distance error correction code, the present invention uses an error correction code C of [n,k,2t+1] _F . The idea of the Hamming distance-based security sketch is to use the error correction code C to correct the data w. For example, input w, uniformly randomly select a codeword c∈C, let s=SS(w)=wc be the required transformation of c to w. Calculate Rec(w',s), obtain c by formula c'=w'-s and then decode c'. Because dis(w,w')≤t, so dis(c,c')≤t. Finally, w is obtained by w=c+s.

2. Fuzzy Extractor

The fuzzy extractor can make two similar data get the same string K. A fuzzy extractor with parameters (M, m, l, t, ε) consists of a pair of efficient generation and regeneration algorithms (KG, REP).

Generating algorithm KG(w)→{K,P}: Take w∈M as input, output an extraction string K∈{0,1} ^l and a public auxiliary string P∈{0,1} ^* .

那么(R,P,E)≈_ε(U,P,E)，则模糊提取器是安全的。The regeneration algorithm REP(w',P)→{K}: takes as input w'∈M and the string P∈{0,1} ^* , if dis(w,w')≤t and KG(w)→{ K,P}, then REP(w',P)=K. If the minimum entropy of M

Then (R,P,E)≈ _ε (U,P,E), then the fuzzy extractor is safe.

Because traditional hash function algorithms (such as SHA-1 or SHA-256) and symmetric encryption algorithms (such as AES-128 or AES-256) cannot be directly applied to achieve secure deduplication of similar data. The invention integrates the similarity preserving hash algorithm (SimHash and PHash), the fuzzy key extraction algorithm based on error correction code, the XOR encryption scheme based on one-time padding and other new technologies, and realizes the safe and efficient similarity in the system of the invention. Data deduplication. The similar data deduplication method can realize the encryption/decryption of customer data, allowing the cloud server to perform secure similar data deduplication on the user's ciphertext data.

The similar data deduplication method is designed to allow the client to encrypt the data, and the cloud server to perform similar duplicate data detection on the ciphertext. In the present invention, the client first applies the similarity preserving hash algorithm to generate the deduplication label and the similarity data key of the plaintext. Using similarity-preserving hashing, similar plaintext data is mapped to similar labels and similar data keys of a certain length (e.g. 64 bits). These fixed-length tags can also significantly reduce storage overhead. The invention designs a random key generation algorithm based on a fuzzy extractor, and similar data within a specific Hamming distance can always obtain the same random encryption key. At this stage, the first user chooses some auxiliary parameters and computes a random key (eg _kw' ) for the plaintext w'. The auxiliary parameters will then be stored on the cloud server. When a subsequent user has a tag tw of similar plaintext data _w (w≈w') and wants to perform a similar data deduplication operation, the cloud server will send auxiliary parameters to the subsequent user, and the subsequent user will run the key regeneration algorithm by Generate the key k _w . If the Hamming distance of file w and file w' is less than a certain value, such as (f _w , f _w' )<t, the key regeneration algorithm will output the same random key k _w' =k _w .

Since the cloud server needs to perform similarity detection on user sensitive data, similar data must be encrypted into similar ciphertext. This would go against the traditional encryption methods employed for message lock encryption. To solve this problem, the present invention uses a simple XOR encryption algorithm based on a one-time padding generator. Similar to stream ciphers, a pseudorandom generator G(·) generates encryption keys of sufficient bit length by using similar keys. If two plaintexts w and _w ' are similar, then their respective similar keys are _kw and _kw' and _kw' =kw; otherwise, _kw' ≠ _kw . The present invention can intuitively obtain

Similar message locking encryption scheme consists of six polynomial time algorithms (FKG, KG, REP, ENC, DEC, TAG):

FKG(1 ^λ ,r ₂ ,w)→fk _w : This is a similarity key generation algorithm based on similarity preserving hash function, which is used to let users calculate the digest information of data. It takes as input a security parameter λ, a random number r ₂ ∈ {0,1} ^λ and a file w, and outputs a similar digest fk _w of a file. Use SimHash or PHash to achieve in actual use.

RKG(1 ^λ ,r ₃ ,fk _w )→{k _w ,P _w }: This is a key generation algorithm that lets users calculate encryption keys and auxiliary parameters for data. x is a public parameter, and the RKG algorithm uses the secure thumbnail sketch algorithm SS{r ₃ ,w}→P _w and the extraction algorithm Ext(w,x)→{K _w } in the fuzzy extractor to generate the auxiliary parameter P={x ,s} and a random encryption key K _w . where r ₃ is a random parameter used to generate a random code C(r ₃ )→c (algorithm C(·) is a code generation algorithm). The code c is used for the SS algorithm in the security sketch.

REP(fk _w' ,P _w )→k _w : This is a key regeneration algorithm, run by the user. Similar to the regeneration algorithm in the fuzzy extractor, by taking the auxiliary parameter Pw and the fuzzy digest _{fkw '} of the file as input, output the private key kw if and only if _{fkw '} is similar to _fkw ; otherwise, output a random value.

其中G(k_w)→{0,1}^|w|是伪随机生成器，以k_w作为输入并输出长度为|w|的伪随机加密密钥G(k_w)。ENC(k _w ,w)→c _w : This is an encryption algorithm run by the user to calculate encrypted data and get the corresponding ciphertext. It takes as input the file w and a private key k _w and returns the ciphertext

where G(k _w )→{0,1} ^|w| is a pseudo-random generator that takes k _w as input and outputs a pseudo-random encryption key G(k _w ) of length |w|.

DEC(kw, _cw )→ _w : This is a decryption algorithm run by the user to compute the plaintext of the input data. It takes as input the ciphertext c _w and a private key k _w and returns the plaintext

TAG(1 ^λ ,r ₁ ,w)→t _w : This is a tag generation algorithm implemented by using a similarity-preserving hash function that can generate the same digest for similar data. The algorithm is run by the user to compute a digest of the input data. It takes as input a security parameter λ, a random number r ₁ and data w and returns a data label _tw .

Based on the definition of the similar data message locking encryption scheme, the present invention provides a schematic diagram of the scheme in FIG. 4 . Similar to message lock encryption, all algorithms may rely on public parameters P _w , which are public to all participants and even adversaries.

The application principle of the present invention will be further described below with reference to specific embodiments.

In the system of the present invention, the present invention assumes that the user Cs is the owner of the data and they wish to outsource their data to be stored on the cloud server and perform similar deduplication storage. After the user uploads the data, the user only needs to keep the identity link (eg ID _w ) and encryption key (eg k _w ) of each data entry (eg data w ). The plaintext data w is obtained by decrypting the ciphertext c _w from the cloud server. The cloud server S stores all data information obtained from the user, and maintains a data set DB={Tag, ID, Cipher}. In the system of the present invention, the data set DB provides three necessary files, that is, the label file, the identity link file and the ciphertext file.

The similar data locking encryption scheme consists of three stages, namely the system establishment stage, the upload stage and the download stage. Since the upload phase and the download phase are two-party interactive protocols, the present invention formally defines the interactive protocol as follows: Π:[P ₁ :in ₁ ; P ₂ :in ₂ ]→[P ₁ :out ₁ ;P ₂ :out ₂ ]. Protocol Π denotes _an interactive protocol run by _two parties P1 and P2, in _i and out _i denote the inputs and outputs of the parties P _i . The detailed structure of the three stages of a similar data locking encryption system is as follows:

和

分别作为算法TAG和算法FKG的输入。)在此之后，用户A运行密钥生成算法RKG(1^λ,r₃,fk_w')→{k_w',P_w'}得到相似加密密钥k_w'和辅助参数P_w'。The system setup phase is run by user C, where r ₁ and r ₂ are two public parameters and r ₃ is a randomly chosen parameter used as input to the [n,k,2t+1] _F error correction code. Without loss of generality, the present invention assumes that user A is the first data owner of data w' and he wishes to upload the data to cloud storage server S. User A first runs the label generation algorithm TAG(1 ^λ ,r ₁ ,w')→t _w' and the similar key generation algorithm FKG(1 ^λ ,r ₂ ,w')→fk _w' to generate the label t of the data w'_w' and similar data summaries fk _w' . (Actually, both the tag generation algorithm TAG and the similar key generation algorithm FKG are implemented with SimHash or PHash, so users will

and

As the input of algorithm TAG and algorithm FKG, respectively. ) After that, user A runs the key generation algorithm RKG(1 ^λ , r ₃ , fk _w' )→{k _w' , P _w' } to obtain the similar encryption key k _w' and auxiliary parameter P _w' .

The upload phase is an interactive protocol that runs between user C and cloud server S. The user C first sends the tag _tw' to the cloud server S, and the tag is used for the server S to perform similar duplication detection in the database it stores. During this phase, two different things happen on the cloud server:

There is no duplicate data. If there is no label _tw similar to the label _tw' in the existing data of the cloud server S, the user needs to upload the data. The upload phase runs the following operations: Upload:[C:t _w' ,w',r ₃ ;s:DedupTb]→[C:k _w' ,c _w' ,P _w' ,Link _w' ;S:t _w' ,R _w' ,c _w' ,Link _w' ].

The user first runs the random key generation algorithm RKG(1 ^λ , r ₃ , fk _w )→{k _w , P _w } to generate random encryption keys and auxiliary parameters. Then encrypt to get the ciphertext ENC(k _w ,w')→c _w' and send {t _w' ,P _w' ,c _w' } to the cloud server S. S stores {t _w' ,P _w' ,c _w' } and returns the link Link _w' to user C for downloading the ciphertext c _w' .

Duplicate data already exists. If the cloud server has stored the data w and the tag tw of the data w is similar to the tag t _{w '} , the upload stage will run the following operations: Upload:[C:t _w' ,w'; s:DedupTb, _Pw' ]→[C:kw, _{cw '} , _Linkw ;S: _Linkw ]. According to the tag _tw' , the cloud server S returns the auxiliary information P _w ={x _w ,s _w } to the user C. When the user receives P _w ={x _w ,s _w }, he first runs the key regeneration algorithm REP(fk _w' ,P _w )→k _w . Then the user encrypts to obtain the ciphertext ENC(k _w ,w')→c _w' of the data w. Afterwards, the cloud server S and the user C execute the similar data ownership proof protocol, and the similar data ownership proof protocol can effectively verify whether the user's ciphertext data c _w' is similar to the data c _w stored on the server. If the user passes the verification, the cloud server returns the user to connect Link _w to download the ciphertext data c _w stored on the cloud server S. Since there is already similar data stored on the cloud server, the user does not need to upload the data c _w' again .

The download phase is an interactive protocol that will be initiated by user C to obtain outsourced data on server S. The protocol is as follows Download: [C:Link _w ,k _w ; s:DedupTb,c _w ]→[C:w; s:⊥]; Intuitively, if user C wants to download the ciphertext of w from server S, the user First, the data identity link Link _w is sent to the server, and the server queries the database DB to find the ciphertext C _{w whose identity link is Link w .} The server then sends the ciphertext C _w to user C. After receiving the ciphertext Cw, user C runs the decryption algorithm DEC( _kw , _cw )→ _w to obtain the plaintext w. In this process, the user first runs the pseudo-random generation algorithm to obtain the decryption key G(k _w ) and calculates the plaintext

后面nmodr个串的长度为

我们首先计算第1块的汉明距离dis_Ham(x₁,y₁)开始，一直计算到第r块的汉明距离dis_Ham(x_r,y_r)，若在计算到第i块时有dis_Ham(x₁,y₁)+...+dis_Ham(x_i,y_i)＞t。则说明两个数据不相似，服务器将不再继续计算之后块的汉明距离。In order to further improve the speed at which the server finds similar tags _tw after receiving tags _tw' , and returns auxiliary information P _w ={x _w ,s _w } to user C, we also design Hamming distance reduction and tag Cutting-optimized methods improve tag query efficiency. The idea of Hamming distance reduction is as follows: Since the cloud server stores a large amount of data and has a large number of labels, if a label tw similar to the label _{tw '} is found by traversing all the labels, it will bring huge computational overhead. Therefore, we designed the 1bits(x) function, which is used to count the number of 1bits in the data x. If the threshold value of similar data is set to t, the two data x and data y are similar and must satisfy -t≤1bits(x)-1bits(y)≤t. We calculate the value of the 1bits(x) function for the tags stored in the cloud server and sort them in descending order. When the server searches for a tag _tw similar to the tag tw', it only searches in the tags that satisfy -t≤1bits( _{tw '} )-1bits( _tw )≤t. After finding a label that satisfies -t≤1bits(t _w' )-1bits(t _w )≤t, we use label cutting optimization to further improve the efficiency of determining the Hamming distance of two data. The principle is to first divide the data into blocks of the same size, and calculate the Hamming distance of each block from front to back. If the Hamming distance of a block is already greater than t, it means that the two data must be different. resemblance. Therefore, we no longer need to calculate the Hamming distance of the specific two data, but when the Hamming distance of the two data has exceeded t when a certain block is calculated, there is no need to continue the calculation and it can be determined that the two data are not similar. In practice, for two data x and data y of length n, we split the data x and data y into (x ₁ ,x ₂ ,...,x _r ) and (y ₁ ,y ₂ ,... .,y _r ). The length of the first r-(nmodr) strings is

The length of the following nmodr strings is

We first calculate the Hamming distance dis _Ham (x ₁ , y ₁ ) of the first block, and continue to calculate the Hamming distance dis _Ham (x _r , y _r ) of the rth block. dis _Ham (x ₁ ,y ₁ )+...+dis _Ham (x _i ,y _i )>t. It means that the two data are not similar, and the server will not continue to calculate the Hamming distance of subsequent blocks.

N是两个大素数的乘积。((N,e)，(N,d))是输出的私钥公钥对。每一个合法的用户首先在密钥服务器进行注册，输入密钥服务器的公钥和明文数据w，选择一个随机数r并通过FKG(r₂,w)算法计算fk_w，然后通过算法RKG(r₃,fk_w)计算得到k_w和P_w。最后用户计算x←H(k_w·r^e)并将x发送给密钥服务器。密钥服务器在收到x之后计算y←x^d modN并将y返回给用户。用户接收到y之后计算z←y·r^-1并验证是否z^emodN＝H(k_w)。如果相等则返回z，如果不相等则返回⊥。z用于通过使用伪随机生成算法计算明文w私有的加密密钥和相似数据验证标签t_w＝h(G(z))。在基于辅助服务器的相似数据消息锁定加密去重方案中，密钥服务器不能获得加密密钥的任何信息。在基于相似标签的相似数据锁定加密去重方案中，每一个数据(比如数据w)的询问标签是通过TAG(1^λ,)→t_w得到的。更准确的来说，

其中g是双线性群的生成元，h是抗碰撞哈希函数，r是随机数。假设用户C拥有数据w'。用户C首先计算fk'←FKG(r₂,w')然后运行密钥再生成算法计算

最后对于每一个记录用户验证云服务器S是否存在

和标签

详细来说，服务器S验证

是否相等。在发现相应的标签之后，用户C与服务器进行数据拥有证明协议。In order to further improve the security of our scheme, we also design a similar data message locking encryption and deduplication scheme based on auxiliary servers and a similar data lock encryption and deduplication scheme based on similar labels. Auxiliary server-based lock-encryption and deduplication scheme for similar data messages resists offline brute force attacks by combining with RSA-based blind signature scheme. Suppose the server of our system uses the RSA key generation algorithm, with parameter e as input, and output N and d such that

N is the product of two large prime numbers. ((N,e), (N,d)) is the output private key and public key pair. Each legitimate user first registers with the key server, enters the public key of the key server and plaintext data w, selects a random number r and calculates fk _w by the FKG(r ₂ ,w) algorithm, and then uses the algorithm RKG(r ₃ , fk _w ) is calculated to obtain k _w and P _w . Finally the user computes x←H( _kw · ^re ) and sends x to the key server. The key server computes y←x ^d modN after receiving x and returns y to the user. The user calculates z←y·r ^-1 after receiving y and verifies whether ^ze modN=H(k _w ). Returns z if equal, ⊥ if not. z is used to verify the label _tw =h(G(z)) by computing a private encryption key and similar data of the plaintext w using a pseudo-random generation algorithm. In a similar data message locking encryption deduplication scheme based on the auxiliary server, the key server cannot obtain any information about the encryption key. In the similar data locking encryption and deduplication scheme based on similar tags, the query tag of each data (such as data w) is obtained by TAG(1 ^λ ,)→t _w . More precisely,

where g is the generator of the bilinear group, h is the collision-resistant hash function, and r is a random number. Suppose user C has data w'. User C first calculates fk'←FKG(r ₂ ,w') and then runs the key regeneration algorithm to calculate

Finally, verify whether the cloud server S exists for each recorded user

and labels

In detail, server S authenticates

are equal. After discovering the corresponding tag, user C conducts a data ownership certification agreement with the server.

The application effect of the present invention will be described in detail below in conjunction with experiments.

The system of the present invention is implemented on the MySQL database system using 3000 lines of C++ code. The present invention utilizes the free GMP library to realize the SimHash algorithm. The password hashing algorithm and the XOR encryption algorithm (SHA-256 and XOR encryption algorithm) are implemented by the OpenSSL library. The present invention runs client and server applications respectively on two computers running Linux 14.04, and the used computer hardware is configured as: 1.70GHz Intel i5-3317U CPU and 4GB memory. In order to conduct experiments on the local area network, the present invention realizes the communication between the client and the server, and places the two machines in the same area. The communication bandwidth of the wired connection between the server and the client is set to 10Mbps. In order to measure the performance of the system of the present invention on real datasets, the present invention uses the Amazon movie review text dataset, which contains 7,911,684 text files, each text file is about 1-15KB in length, and the image data exceeds 14 million images .

Table 1 Computation time of different algorithms

The test results of similar data locking encryption systems are shown in Table 1. The specific conditions are as follows: FKG is a similar hash function, and the present invention adopts 64-bit SimHash and 64-bit PHash for implementation in this paper (SimHash can only be used to process text files, and PHash can be applied to text and image files). Given fixed-length text data (1KB), the average computation time for SimHash and PHash is 1386us and 5312us, respectively. Given an image JPEG data (10KB), the average computation time of PHash is 6439us. The RKG algorithm needs 261us and 885us to calculate at 64 and 256-bit lengths, respectively. The REP algorithm takes 124us and 368us at 64 and 256-bit lengths, respectively. The data in ENC and DEC are encrypted and decrypted by XOR operation, and it takes 338us to perform ENC and DEC operations on a 1KB bit string. Similar to FKG, the implementation of TAG is also achieved by using a similar hash function (ie the 64-bit SimHash algorithm used in the scheme of the present invention). In addition, the present invention observes that the calculation time of FKG, ENC, DEC and TAG is related to the size of the input, and the present invention presents the simulation results in FIG. 5 and FIG. 6 respectively. Their computation time increases linearly as the size of the input data increases.

Finally, the present invention tests the similar data deduplication system of the present invention with text data and image data. As shown in Figure 7, the experiments were run on a text database of 100,000 records, each with a 256-bit tag. As shown in Figure 8, the experiments were run on an image database of 1000 records, each with 64-bit labels. In both tests, the plaintext client will interact with the server to verify that a similar ciphertext exists in the database. If there are no duplicates of similar data in the server database, the client will upload its ciphertext. Otherwise, the server will de-duplicate similar data with the user, and send a link to the similar data to the user.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (3)

1. A locking, encrypting and de-duplicating method for similar data messages in a cloud environment is characterized in that the locking, encrypting and de-duplicating method for similar data messages in the cloud environment adopts a similarity preserving hash function, and realizes the de-duplication of similar data by a secret key extraction method based on an error correcting code and a secure symmetric encryption algorithm based on a pseudo-random generator; the label query efficiency is improved by a Hamming distance reduction and label cutting optimization method;

a client of the similar data message locking encryption deduplication method in the cloud environment applies a similarity preserving hash algorithm to generate a deduplication label of a plaintext and a similar data key; using the similarity-preserving hash, similar plaintext data will be mapped to similar labels and similar data keys having a particular length; the same random encryption key can be always obtained from similar data within a specific Hamming distance, and a first user selects some auxiliary parameters and calculates the random key of a plaintext w'; the auxiliary parameters are stored on the cloud server; when the subsequent user has a label t of similar plaintext data w (w ≈ w')/t_wAnd when the similar data deduplication operation is required to be executed, the cloud server sends the auxiliary parameters to the later uploaded user, and the later uploaded user generates the key k by running a key regeneration algorithm_w(ii) a If the Hamming distance of file w and file w' is less than a specified value, if (f)_w,f_w') If t is less than t, the key regeneration algorithm outputs the same random key k_w'＝k_w；

The similar message locking encryption scheme of the similar data message locking encryption deduplication method in the cloud environment is formed by six polynomial time algorithms (FKG, KG, REP, ENC, DEC, TAG):

FKG(1^λ,r₂,w)→fk_w: the method is a similar key generation algorithm based on a similar reserved hash function and is used for enabling a user to calculate summary information of data; with a security parameter lambda, a random number r₂∈{0,1}^λThe similar abstract fk of a file is output by taking the file w as input_w；

RKG(1^λ,r₃,fk_w)→{k_w,P_w}: is a key generation algorithm for the user to calculate the encryption key and auxiliary parameters of the data; x is a public parameter, RKG algorithm uses the outline algorithm SS r of the safety outline₃,w}→P_wAnd the extraction algorithm Ext (w, x) → { K ] in the blur extractor_wGenerating auxiliary parameters P ═ x, s and a random encryption key K_wWherein r is₃Is a random parameter for generating a random code C (r)₃) Algorithm C (-) is a code generation algorithm, code C is used for the SS algorithm in the safety sketch;

REP(fk_w',P_w)→k_w: is a key regeneration algorithm, which is run by the user by applying the auxiliary parameter P_wAnd fuzzy summary fk of the file_w'As input, if and only if fk_w'And fk_wOutputting the private key k at similar times_w(ii) a Otherwise, outputting a random value;

ENC(k_w,w)→c_w: is an encryption algorithm operated by user to calculate encrypted data and obtain corresponding ciphertext, a file w and a private key k_wReturning as input ciphertext

Wherein G (k)_w)→{0,1}^|w|Is a pseudo-random generator, with k_wAs input and output a pseudorandom encryption key G (k) of length | w |_w)；

DEC(k_w,c_w) → w: is a decryption algorithm run by the user to calculate the plaintext of the input data; it uses the ciphertext c_wAnd a private key k_wReturning as input the plaintext

TAG(1^λ,r₁,w)→t_w: is a label generation algorithm, realized by using a similar retention hash function, operated by a user to calculate the abstract of input data; with a security parameter lambda, a random number r₁And data w as input, return data tag t_w。

2. The method for locking encryption and de-duplication of similar data messages in cloud environment according to claim 1, wherein the method for locking encryption and de-duplication of similar data messages in cloud environment comprises the following steps:

the client generates a duplicate removal label of a plaintext by using a similarity retention hash algorithm and sends the duplicate removal label to a cloud server, and the cloud server judges whether similar data are stored on the cloud server;

if the cloud server does not have similar data, the user is required to generate a similar data key and auxiliary information for similar key recovery, and encrypted ciphertext data and the auxiliary information are sent to the cloud server;

if the cloud server has similar data, returning auxiliary information for recovering the similar key to the user, encrypting the data by the user through the recovered similar key, and performing similar data ownership verification by using the obtained ciphertext and the server, wherein if the data is verified, the cloud server allows the user to access the data.

3. The method for locking encryption and de-duplication of similar data messages in cloud environment as claimed in claim 1, wherein the method for locking encryption and de-duplication of similar data messages in cloud environment uses [ n, k,2t +1 ]]_FThe idea of the hamming distance-based safety sketch is to use an error correction codeThe method comprises the steps of correcting error of data w, inputting w, uniformly and randomly selecting a code word C ∈ C, enabling s to be SS (w) to be w-C to be the transformation needed from C to w, calculating Rec (w ', s), obtaining C through a formula C' to w '-s and then decoding C', and obtaining w through w to C + s.

Images