CA2936106A1 - Encrypted data - data integrity verification and auditing system - Google Patents

Encrypted data - data integrity verification and auditing system Download PDF

Info

Publication number
CA2936106A1
CA2936106A1 CA2936106A CA2936106A CA2936106A1 CA 2936106 A1 CA2936106 A1 CA 2936106A1 CA 2936106 A CA2936106 A CA 2936106A CA 2936106 A CA2936106 A CA 2936106A CA 2936106 A1 CA2936106 A1 CA 2936106A1
Authority
CA
Canada
Prior art keywords
data
encrypted
audit
auditing
search
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
CA2936106A
Other languages
French (fr)
Inventor
Mirza Kamaludeen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CA2936106A priority Critical patent/CA2936106A1/en
Publication of CA2936106A1 publication Critical patent/CA2936106A1/en
Abandoned legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/64Protecting data integrity, e.g. using checksums, certificates or signatures

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Bioethics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Storage Device Security (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

There is provided methods and systems suitable for ensuring data integrity in encrypted data.
Potential applications include, but are not limited to, secure and remote storage systems such as email servers and cloud servers, encrypted devices such as hard drives, usb flash drives and smart cards. The proposed solution combines the use of symmetric encryption and encrypted indices to provide searching and auditing capabilities. The entire auditing process is executed in the encrypted domain.
This system provides assurance that the encrypted files were not modified or tampered with, and also incorporates search and contiguous search capabilities over the encrypted files. One such compatible system would be the private scanning solution outlined in "Encrypted Data - Computer Virus and Malware Detection System."
Our approach involves the data owner encrypting and sending the data files and their indices, in accordance with an encryption scheme, to a storage repository, device, or cloud storage servers. Then, with or without the use of a third party auditor, the data owner can verify the integrity of the data set by performing random checks on any parts of the data set that is indexed.
2.0 FIELD OF INVENTION
This disclosure relates to data integrity assurance and auditing for encrypted storage systems.

Description

ENCRYPTED DATA - DATA INTEGRITY
VERIFICATION AND AUDITING SYSTEM
1.0 BACKGROUND OF INVENTION
Cloud services provide a low cost and flexible solution to companies for managing computing resources. However, the out-sourcing of data also raises various security and privacy concerns.
In recent years, many researchers have investigated solutions to various issues surrounding cloud storage services and improvements are continually being made. While there are significant efforts in addressing the individual problems in the area, few have looked at more complete solutions, combining the various desired functionalities. In this disclosure, we describe a solution for storing, auditing and searching through encrypted data sets on cloud storage. Instead of including resources that enable the functionalities separately, our setup uses the same pool of resources, leading to a smaller overhead than simply using two separate solutions. Our solution maintains many desirable features found in search and audit schemes such as privacy of data and search terms, proof of retrievability (POR) with theoretically unbounded number of audits and public verifiability.
While solutions abound for processing unencrypted data, privacy aware solutions for encrypted data has mostly been restricted to research and academic studies. The traditional use of SQL for search has mostly targeted data in plain where computation and advanced search such as COUNT and VALUE > X are relatively easy to implement. The need for data integrity assurance has mostly been on identifying and recovering from data corruption, such as the use of checksums, error correction codes and RAID mechanisms. The problem of integrity assurance in outsourced data, especially when encrypted for privacy reasons, has yet to be tackled by the industry. In particular, when data is stored in a remote server that is not under the control of the data owner, all previous solutions would fail to ensure data integrity as the remote server could simply respond with a correct checksum or a "no error detected" even when it does not store the data itself. Since the owner does not know the content of the data, mechanism needs to be in place to ensure the data is stored and unmodified.
2.0 SUMMARY OF INVENTION
We propose a solution for performing data integrity auditing on encrypted data. The solution works with any search and contiguous search scheme, such as a keyword and phrase search scheme, that has the ability to query a keyword/block's location within a file. The data set should also use a symmetric encryption algorithm. Running in counter mode would provide the best efficiency. We describe the systems and mechanisms for ensuring data integrity in both privately accessed remote storage and where a third party auditor is employed, which we termed the public storage setting.
One embodiment of the invention contains the following:
- An encrypted index mapping keywords/blocks to files - Encrypted indices mapping keyword/blocks to their locations within each file Due to reuse of resources, our auditing solution achieves performance comparable to leading keyword and phrase search algorithms.
3.0 DESCRIPTION
For private storage and auditing, data is first indexed to create multiple indexes. One set of indices maps data blocks to files over multiple files, while the other set maps block locations to data within individual files. The data and the indices are then encrypted using a standard encryption algorithm in the encryptor. To perform an audit, the user initiates a search for a random set of blocks. The storage server performs the search and returns the encrypted results to the user, who decrypts the information to determine set of ciphertexts and their corresponding locations. The user then selects a subset of ciphertexts and generates an audit message, specifying a bit sequence and requesting its hash signature. The storage server responds to the audit message with the requested hash signature. The user verifies and is assured of the integrity of the data if the signature matches.
For secure storage with third party (public) auditing, data is first indexed to create multiple indexes. One set of indices maps data blocks to files over multiple files, while the other set maps block locations to data within individual files. The data and the indices are then encrypted using a standard encryption algorithm in the encryptor. A set of audit messages with corresponding hash signatures are also generated based on the encrypted data and, optionally, the indices. The set of audit messages and hash signatures are sent to the third party auditor.
The encrypted data and indices are uploaded to the storage server. To perform an audit, the third party auditor selects and sends an audit message to the storage server.
The storage server responds to the audit message with the requested hash signature. The auditor verifies and is assured of the integrity of the data if the signature matches the corresponding one provided by the user.
When using a third party auditor, mechanism can be provided such that the user may generate and provide the auditor with a new set of audit messages and hash signatures in the event where the initial set of audit messages and corresponding hash signatures were exhausted.
Should a new set be required, the user initiates a search for random sets of blocks. The storage server performs the search and returns the encrypted results to the user, who decrypts the information to determine sets of ciphertexts and their corresponding locations. The user then selects subsets of ciphertexts and generates a set of audit messages and their corresponding hash signatures. The set of audit messages and signatures are sent to the third party auditor, who may continue auditing the storage server using the new set of messages.

One embodiment of the method for private auditing implemented on a Cloud Storage system is:
For setup, a search system using encrypted indices is used:
1) Parse all files in data set for blocks of size n to generate an index A, mapping blocks to files 2) Parse each file for blocks of size n to generate an index B,, mapping blocks to block location, where each row also includes the encrypted row number and the encrypted block 3) Encrypt all files and indices using a symmetric encryption scheme in counter mode
4) Maintain a record of the number of rows in each index, e.g. i=1, n1= 215;
i=2, n2 = 842...
5) Upload files and indices to the cloud server For auditing,
6) Perform a search query for random blocks or rows in randomly selected files
7) Select nA bits randomly for audit based on decrypted query results. Since blocks in plaintext and their locations can be obtained from the decrypted index entries, their ciphertexts in counter mode can be computed to generate the hash signature of the n A bits.
8) Generate the audit message for the nA selected bits, specifying file ID's, run length, byte locations and a random salt value of size n, (i.e. n, randomly selected bits)
9) Storage server responds to the audit by computing the hash signature of the specified bits from the stored encrypted data
10) Verify that the signatures match One embodiment of the method for public auditing, where a third party auditor is employed, implemented on a Cloud Storage system is:

For setup, a search system using encrypted indices is used:
1) Parse all files in data set for blocks of size n to generate an index A, mapping blocks to files 2) Parse each file for blocks of size n to generate an index B,, mapping blocks to block location, where each row also includes the encrypted row number and the encrypted block 3) Encrypt all files and indices using a symmetric encryption scheme in counter mode 4) Maintain a record of the number of rows in each index, e.g. 1=1, n1 = 215;
i=2, nz = 842...
5) Randomly selects sets of nA bits from the encrypted data to generate t audit messages and their corresponding hash signatures. Each audit message includes the file ID's, run length and byte locations of the nA bits under audit in addition to a random salt value of size n, 6) Sends the set of audit messages and corresponding hash signatures to auditor 7) Upload files and indices to the cloud server For auditing, 8) Auditor randomly selects an audit message from the list provided by data owner and sends it to the storage server.
9) The expected hash signature is placed in memory and the audit message is removed from the list.
10) Storage server responds to the audit by computing the hash signature of the specified bits from the stored encrypted data
11) Auditor verifies whether the signatures match Should the auditor used up all the audit messages available, the data owner is notified. If desired, the data owner can provide the auditor a new set of messages using the following protocol:
1) Perform a search query for random blocks or rows in randomly selected files 2) Select sets of n A bits randomly for audit based on decrypted query results. Since blocks in plaintext and their locations can be obtained from the decrypted index entries, their ciphertexts in counter mode can be computed to generate the hash signature of the nA bits.
3) Generate audit messages for the sets of nA selected bits, specifying file ID's, run length, byte locations and a random salt value 4) Send the new set of audit messages and their corresponding hash signatures to the auditor

Claims (43)

1.0 CLAIMS
The Following claims and scope of claims should not be limited by examples provided herein, but should be given the broadest interpretation consistent with the description as a whole.
1. A method of verifying data integrity and performing auditing in encrypted data, where a searching system over encrypted data, e.g. using encrypting indices over a data set, is available. The method comprises:
- Setup for the searching system, e.g. multiple index generation and encryption - Searching for sets of data - Selecting a subset for audit and computing the corresponding hash signature - Generating an audit message specifying the ciphertext bit sequence locations - Generating a signature for the requested bit sequence from the stored encrypted data - Verifying that the signatures match
2. A method according to claim 1 where the data is encrypted using a symmetric block cipher in cipher block chaining or counter mode
3. A method according to claim 1 where the data is encrypted using a stream cipher
4. A method according to claim 1 where audit messages are pre-generated during setup and stored locally by the data owner/user to improve efficiency at the cost of local storage
5. A method according to claim 1 where a salt, e.g. a random bit sequence, is added to the bit sequence under audit to protect against replay attack
6. A method according to claim 1 where multiple indices are used to enable multi-level scanning, as a trade-off between privacy and efficiency
7. A method according to claim 1 where multiple searching systems are used to support the auditing functionality, e.g. row based search and keyword search
8. A method according to claim 1 where audit messages are generated by the data owner who passively collects search data requested by users, such as a publicly searched encrypted database
9. A method of claim 1 where resources allocated for the purpose of search is further processed by a proof-of-retrievability system, e.g. Indices are encoded by a erasure code
10. A method of claim 1 where the audit message specifies a run length, document ID's and bit/byte locations, such that the number of bits/bytes equal to the run length at the bit/byte locations of document ID's correspond to the bit sequence under audit
11. A method of claim 1 where bloom filters are used to provide the required search functionality
12. A method of claim 1 for verifying data integrity and auditing without transporting the encrypted files them selves
13. A system of claim 1 for verifying data integrity and auditing without transporting the encrypted files them selves
14. A method of verifying data integrity and auditing in encrypted data, employing a third party public auditor. The method comprises:
- Setup for the searching system, e.g. multiple index generation and encryption - Generating a set of audit messages specifying ciphertext bit sequence locations and the set of corresponding hash signatures - Sending the set of audit messages and hash signatures to auditor - Auditor initiates auditing by sending an audit message to the storage provider - Storage provider responds by sending the requested signature for the bit sequence - Verifying that the signatures match by the auditor
15. A method according to claim 14 where the data is encrypted using a symmetric block cipher in cipher block chaining or counter mode
16. A method according to claim 14 where the data is encrypted using a stream cipher
17. A method according to claim 14 where a salt, e.g. a random bit sequence, is added to the bit sequence under audit to protect against replay attack
18. A method according to claim 14 where multiple indices are used to enable multi-level scanning, as a trade-off between privacy and efficiency
19. A method according to claim 14 where multiple searching systems are used to support the auditing functionality, e.g. row based search and keyword search
20. A method according to claim 14 where audit messages are generated by the data owner who passively collects search data requested by users, such as a publicly searched encrypted database
21. A method of claim 14 where resources allocated for the purpose of search is further processed by a proof-of-retrievability system, e.g. Indices are encoded by a erasure code
22. A method of claim 14 where the audit message specifies a run length, document ID's and bit/byte locations, such that the number of bits/bytes equal to the run length at the bit/byte locations of document ID's correspond to the bit sequence under audit
23. A method of claim 14 where bloom filters are used to provide the required search functionality
24. A method of claim 14 where the audit messages or the hash signatures are stored by the data owner instead of by the auditor, and where auditing is performed in a three-way protocol involving both auditor and data owner
25. A method of claim 14 for verifying data integrity and auditing without transporting the encrypted files them selves
26. A system of claim 14 for verifying data integrity and auditing without transporting the encrypted files them selves
27. A method of claim 14 where a method is in place to replenish the list of audit messages and hash signatures for the auditor. The method comprises:
- Searching for sets of data - Selecting subsets for audit and computing the corresponding hash signatures - Generating audit messages specifying ciphertext bit sequence locations - Sending the set of audit messages with corresponding hash signatures to auditor
28. A method according to claim 27 where the data is encrypted using a symmetric block cipher in cipher block chaining or counter mode
29. A method according to claim 27 where the data is encrypted using a stream cipher
30. A method according to claim 27 where a salt, e.g. a random bit sequence, is added to the bit sequence under audit to protect against replay attack
31. A method according to claim 27 where multiple indices are used to enable multi-level scanning, as a trade-off between privacy and efficiency
32. A method according to claim 27 where multiple searching systems are used to support the auditing functionality, e.g. row based search and keyword search
33. A method according to claim 27 where audit messages are generated by the data owner who passively collects search data requested by users, such as a publicly searched encrypted database
34. A method of claim 27 where resources allocated for the purpose of search is further processed by a proof-of-retrievability system, e.g. Indices are encoded by a erasure code
35. A method of claim 27 where the audit message specifies a run length, document ID's and bit/byte locations, such that the number of bits/bytes equal to the run length at the bit/byte locations of document ID's correspond to the bit sequence under audit
36. A method of claim 27 where bloom filters are used to provide the required search functionality
37. A method of claim 27 where the audit messages or the hash signatures are stored by the data owner instead of by the auditor, and where auditing is performed in a three-way protocol involving both auditor and data owner
38. A method of claim 27 for verifying data integrity and auditing without transporting the encrypted files them selves
39. A system of claim 27 for verifying data integrity and auditing without transporting the encrypted files them selves
40. A cloud based data integrity and auditing system pertaining to claim 1, 14 and 27 for encrypted data
41. A network based data integrity and auditing system pertaining to claim 1, 14 and 27 for encrypted data
42. A device based data integrity and auditing system pertaining to claim 1, 14 and 27 for encrypted data
43. A method or system pertaining to claim 1, 14 and 27 implemented as an embedded integrated circuit
CA2936106A 2016-07-14 2016-07-14 Encrypted data - data integrity verification and auditing system Abandoned CA2936106A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CA2936106A CA2936106A1 (en) 2016-07-14 2016-07-14 Encrypted data - data integrity verification and auditing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CA2936106A CA2936106A1 (en) 2016-07-14 2016-07-14 Encrypted data - data integrity verification and auditing system

Publications (1)

Publication Number Publication Date
CA2936106A1 true CA2936106A1 (en) 2018-01-14

Family

ID=60989363

Family Applications (1)

Application Number Title Priority Date Filing Date
CA2936106A Abandoned CA2936106A1 (en) 2016-07-14 2016-07-14 Encrypted data - data integrity verification and auditing system

Country Status (1)

Country Link
CA (1) CA2936106A1 (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3591563A1 (en) * 2018-07-04 2020-01-08 Koninklijke Philips N.V. Verifying the integrity of a plurality of data blocks
CN110673873A (en) * 2019-10-09 2020-01-10 成都安恒信息技术有限公司 Audit-based software release method
CN111339570A (en) * 2020-02-27 2020-06-26 青岛大学 Method, device, equipment and medium for verifying integrity of cloud storage file
CN111611625A (en) * 2020-05-26 2020-09-01 牛津(海南)区块链研究院有限公司 Cloud data integrity auditing method and device and computer readable storage medium
CN111859412A (en) * 2020-06-11 2020-10-30 中国科学院信息工程研究所 Cloud data position public verification method and system based on CPOR model
CN112887281A (en) * 2021-01-13 2021-06-01 西安电子科技大学 Storage method and system supporting efficient audit and multi-backup ciphertext deduplication and application
CN113364600A (en) * 2021-08-11 2021-09-07 西南石油大学 Certificateless public auditing method for integrity of cloud storage data
CN113656840A (en) * 2021-07-16 2021-11-16 北京航空航天大学杭州创新研究院 Dynamic integrity verification method with accountability
CN113660258A (en) * 2021-08-13 2021-11-16 重庆中科云从科技有限公司 Method, system, medium, and apparatus for preventing file from being tampered during transmission
CN114201773A (en) * 2021-12-13 2022-03-18 合肥工业大学 SkNN query method and system supporting access time limitation and result verification
WO2023020429A1 (en) * 2021-08-20 2023-02-23 清华大学 Data auditing method and apparatus, and storage medium

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3591563A1 (en) * 2018-07-04 2020-01-08 Koninklijke Philips N.V. Verifying the integrity of a plurality of data blocks
CN110673873B (en) * 2019-10-09 2022-11-01 成都安恒信息技术有限公司 Audit-based software release method
CN110673873A (en) * 2019-10-09 2020-01-10 成都安恒信息技术有限公司 Audit-based software release method
CN111339570A (en) * 2020-02-27 2020-06-26 青岛大学 Method, device, equipment and medium for verifying integrity of cloud storage file
CN111339570B (en) * 2020-02-27 2022-12-02 青岛大学 Method, device, equipment and medium for verifying integrity of cloud storage file
CN111611625A (en) * 2020-05-26 2020-09-01 牛津(海南)区块链研究院有限公司 Cloud data integrity auditing method and device and computer readable storage medium
CN111611625B (en) * 2020-05-26 2023-04-07 牛津(海南)区块链研究院有限公司 Cloud data integrity auditing method and device and computer readable storage medium
CN111859412A (en) * 2020-06-11 2020-10-30 中国科学院信息工程研究所 Cloud data position public verification method and system based on CPOR model
CN112887281A (en) * 2021-01-13 2021-06-01 西安电子科技大学 Storage method and system supporting efficient audit and multi-backup ciphertext deduplication and application
CN113656840A (en) * 2021-07-16 2021-11-16 北京航空航天大学杭州创新研究院 Dynamic integrity verification method with accountability
CN113656840B (en) * 2021-07-16 2024-01-02 北京航空航天大学杭州创新研究院 Dynamic integrity verification method with accountability
CN113364600B (en) * 2021-08-11 2021-12-07 西南石油大学 Certificateless public auditing method for integrity of cloud storage data
CN113364600A (en) * 2021-08-11 2021-09-07 西南石油大学 Certificateless public auditing method for integrity of cloud storage data
CN113660258A (en) * 2021-08-13 2021-11-16 重庆中科云从科技有限公司 Method, system, medium, and apparatus for preventing file from being tampered during transmission
WO2023020429A1 (en) * 2021-08-20 2023-02-23 清华大学 Data auditing method and apparatus, and storage medium
CN114201773A (en) * 2021-12-13 2022-03-18 合肥工业大学 SkNN query method and system supporting access time limitation and result verification
CN114201773B (en) * 2021-12-13 2024-02-13 合肥工业大学 SkNN query method and system supporting access time limitation and verifiable result

Similar Documents

Publication Publication Date Title
CA2936106A1 (en) Encrypted data - data integrity verification and auditing system
AU2018367363B2 (en) Processing data queries in a logically sharded data store
US11381398B2 (en) Method for re-keying an encrypted data file
US20210099287A1 (en) Cryptographic key generation for logically sharded data stores
US9977918B2 (en) Method and system for verifiable searchable symmetric encryption
JP6306077B2 (en) Community-based deduplication of encrypted data
JP5777630B2 (en) Method and apparatus for document sharing
US9122888B2 (en) System and method to create resilient site master-key for automated access
KR101371608B1 (en) Database Management System and Encrypting Method thereof
CN111130757A (en) Multi-cloud CP-ABE access control method based on block chain
US9602280B2 (en) System and method for content encryption in a key/value store
US20100215175A1 (en) Methods and systems for stripe blind encryption
AU2017440029B2 (en) Cryptographic key generation for logically sharded data stores
CN115225409B (en) Cloud data safety duplicate removal method based on multi-backup joint verification
Giri et al. A survey on data integrity techniques in cloud computing
Nithiavathy Data integrity and data dynamics with secure storage service in cloud
US8364979B1 (en) Apparatus, system, and method to efficiently search and modify information stored on remote servers, while hiding access patterns
Rashid et al. Secure enterprise data deduplication in the cloud
Bindu et al. Secure data storage in cloud computing
Suthar et al. EncryScation: A novel framework for cloud iaas, daas security using encryption and obfuscation techniques
Sowparnika et al. Improving data integrity on cloud storage services
Shanmugakani et al. An explicit integrity verification scheme for cloud distributed systems
Venkatesh et al. Secure authorised deduplication by using hybrid cloud approach
CN113014567B (en) Internet IP message data verification processing method, device and system
Parveen et al. Cloud Dynamic and Public Auditing Scheme for Secure Data by using RSA with Modified Dynamic Hash Table

Legal Events

Date Code Title Description
FZDE Dead

Effective date: 20190208