NZ782231B2 - Method and system for distributed data storage with enhanced security, resilience, and control - Google Patents

Method and system for distributed data storage with enhanced security, resilience, and control Download PDF

Info

Publication number
NZ782231B2
NZ782231B2 NZ782231A NZ78223120A NZ782231B2 NZ 782231 B2 NZ782231 B2 NZ 782231B2 NZ 782231 A NZ782231 A NZ 782231A NZ 78223120 A NZ78223120 A NZ 78223120A NZ 782231 B2 NZ782231 B2 NZ 782231B2
Authority
NZ
New Zealand
Prior art keywords
shards
metadata
key
cryptography
encrypting
Prior art date
Application number
NZ782231A
Other versions
NZ782231A (en
Inventor
Jaeyoon Chung
Original Assignee
Myota Inc
Filing date
Publication date
Application filed by Myota Inc filed Critical Myota Inc
Priority claimed from PCT/US2020/032781 external-priority patent/WO2020236500A1/en
Publication of NZ782231A publication Critical patent/NZ782231A/en
Publication of NZ782231B2 publication Critical patent/NZ782231B2/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/164—File meta data generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/178—Techniques for file synchronisation in file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/10—Protecting distributed programs or content, e.g. vending or licensing of copyrighted material ; Digital rights management [DRM]
    • G06F21/107—License processing; Key processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/602—Providing cryptographic facilities or services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6209—Protecting access to data via a platform, e.g. using keys or access control rules to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0658—Controller construction arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/544—Buffers; Shared memory; Pipes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
    • H04L9/0618—Block ciphers, i.e. encrypting groups of characters of a plain text message using fixed encryption transformation
    • H04L9/0631—Substitution permutation network [SPN], i.e. cipher composed of a number of stages or rounds each involving linear and nonlinear transformations, e.g. AES algorithms
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/08—Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords
    • H04L9/0816—Key establishment, i.e. cryptographic processes or cryptographic protocols whereby a shared secret becomes available to two or more parties, for subsequent use
    • H04L9/0819—Key transport or distribution, i.e. key establishment techniques where one party creates or otherwise obtains a secret value, and securely transfers it to the other(s)
    • H04L9/0822—Key transport or distribution, i.e. key establishment techniques where one party creates or otherwise obtains a secret value, and securely transfers it to the other(s) using key encryption key
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/08—Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords
    • H04L9/0816—Key establishment, i.e. cryptographic processes or cryptographic protocols whereby a shared secret becomes available to two or more parties, for subsequent use
    • H04L9/085—Secret sharing or secret splitting, e.g. threshold schemes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/08—Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords
    • H04L9/0894—Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/14—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols using a plurality of keys or algorithms
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
    • H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
    • H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/50—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols using hash chains, e.g. blockchains or hash trees

Abstract

method and system for encrypting and reconstructing data files, using computational and theoretical cryptography, including related metadata, is disclosed. The method involves separately encrypting data and metadata as chaining processes and integrating the chaining processes together with strategic storage distribution techniques and parsing techniques which results in the integrated benefits of the collection of techniques. As disclosed, the content data is separated from its metadata. Encryption keys are embedded in the metadata. Content data and metadata are stored separately and, in a flexible, distributed, and efficient manner, at least in part to assure improved resiliency. In addition, the processes are preferably implemented locally, including at the site of the content data or at a proxy server.

Description

A method and system for encrypting and reconstructing data files, using computational and theoretical cryptography, including related metadata, is disclosed. The method involves separately encrypting data and metadata as chaining processes and integrating the chaining processes together with strategic storage distribution techniques and parsing techniques which results in the integrated benefits of the collection of techniques. As disclosed, the content data is separated from its metadata. Encryption keys are embedded in the metadata. Content data and metadata are stored separately and, in a flexible, distributed, and efficient manner, at least in part to assure improved resiliency. In addition, the processes are preferably implemented locally, including at the site of the content data or at a proxy server.
NZ 782231 B2 METHOD AND SYSTEM FOR DISTRIBUTED DATA STORAGE WITH ENHANCED SECURITY, RESILIENCE, AND CONTROL Cross-Reference to Related Application id="p-1" id="p-1"
[0001] This application claims priority to U.S. Provisional Patent Application No. 62/851,146, filed May 22, 2019, and now pending, the entirety of which is incorporated by reference.
Background of the Present Invention id="p-2" id="p-2"
[0002] Protecting data is a well-known problem in the storage technology area in terms of security and resilience. There are well-known solutions such as Erasure Code, which is widely used for CDs, DVDs, QR Codes, etc., to improve the ability of error correction over prior solutions, and Shamir’s Secret Sharing Scheme (SSSS) which protects a secret with a polynomial interpolation technique. Their (t, n) threshold property requires at least t data pieces called shares (or shards) from n to reconstruct original data. Like n replicated copies, but introducing an additional constraint t, this property improves data resilience when reconstructing the original because it allows n-t storage node failures without service interruption. From the perspective of data protection, the (t, n) threshold property also reveals the original data only if at least t shares are accessible and valid. id="p-3" id="p-3"
[0003] Erasure Code has a goal of correcting bit errors within the data with maximizing transmission or storage efficiency. Thus, most applications are based solely on Erasure Code such as Reed-Solomon (RS) Code. In computer storage, Erasure Code has been used to implement Redundant Array of Independent Disks (RAID), specifically levels 5 and 6, which are designed for a reliable storage component under different levels of failures. id="p-4" id="p-4"
[0004] A large-scale data storage system causes a new technical challenge, i.e., managing and protecting metadata. To achieve flexibility and scalability, data is stored to distributed storages along with its metadata, where the metadata includes information regarding where the required data pieces are located. Thus, to store metadata reliably and securely, another layer of data protection is ordinarily necessary. id="p-5" id="p-5"
[0005] For example, Shamir’s Secret Sharing Scheme (SSSS) and RS have been used to respectively protect security and error correction of data, even though SSSS and RS Code have the (t, n) threshold property, which requires at least t data shares from n to reconstruct original data. They aim at cryptography and error correction respectively.
SSSS is designed as a cryptography technique that stores a secret into multiple shares, n, without use of an encryption key. SSSS leverages polynomial interpolation which guarantees theoretical cryptography, so no methodology has been known to break SSSS with less than t shares.
RS Code also has the same (t, n) threshold property, but is designed for error correction and storage efficiency. Unlike SSSS, RS Code uses a linear mapping such that

Claims (9)

1. A method for a processor to encrypt at least one computer file based on a combination of computational and theoretical cryptography, said computer file accessible at least on a local device, said computer file including a content data portion and an associated metadata portion, 5 comprising the steps of: generating a plurality of randomly generated encryption keys; selecting at least one computer file for encryption; parsing said content data portion of said computer file into a chain of content chunks, each said chunk assigned a chunk ID; 10 using computational cryptography, said computational cryptography includes use of one or more encryption algorithms and erasure coding, using said at least one encryption key per chunk, encrypting each of said content chunks; using computational cryptography encoding and parsing each of said content chunks into a plurality of content shards; 15 using theoretical cryptography, without the need for an encryption key said theoretical cryptography including secret sharing methods and storing a secret in multiple shards, n, encrypting said chunk IDs; augmenting said metadata portion with said encrypted chunk IDs thereby forming an augmented metadata portion; 20 parsing said plurality of randomly generated encryption keys into a plurality of key shards; using theoretical cryptography, encrypting said plurality of key shards; adding said encrypted plurality of key shards into said augmented metadata portion; parsing a subset of said metadata portion into a plurality of metadata shards; encrypting said metadata shards; 25 delivering said plurality of content shards to at least a first storage location; and delivering said plurality of metadata shards to at least a second storage location; wherein said at least a first storage location differs from said at least a second storage location, said method is configured to protect stored data from brute force attacks, and said method is configured such that decryption requires knowledge of t out of n content shards, t out 30 of n key shards, and t out of n metadata shards, where t and n are integers; and wherein where t is a number of required shards to reconstruct and n is a number of shards stored, parameters t and n of metadata shards, key shards, and data shards are each independently configurable, individually selectable by a user, and where t is an integer greater than 1 and n is an integer greater than t. 35
2. The method of claim 1, where the encrypted key shards and said chunk IDs are separately stored.
3. The method of either claim 1 or claim2, where the steps of parsing said at least one key into a plurality of key shards and encrypting said plurality of key shards is at least partially performed using Shamir's Secret Sharing Scheme (SSSS).
4. The method of any one of claims 1 to 3, where the computational cryptography portion of 5 the method includes use of Reed-Solomon encoding.
5. The method of any one of claims 1 to 4, where the step of encrypting each of said content chunks includes use of AES-256.
6. The method of any one of claims 1 to 5, where said content data portion is fully encrypted before encoding. 10
7. The method of any one of claims 1 to 6, where at least one file attribute in said metadata portion is not encrypted.
8. The method of any one of claims 1 to 7, where at least some of said metadata portion is stored in a vault on said local device.
9. The method of any one of claims 1 to 8, where a number, n, of each of metadata 15 storage, key storage, and data storage is configurable and each is greater than 2.
NZ782231A 2020-05-14 Method and system for distributed data storage with enhanced security, resilience, and control NZ782231B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962851146P 2019-05-22 2019-05-22
PCT/US2020/032781 WO2020236500A1 (en) 2019-05-22 2020-05-14 Method and system for distributed data storage with enhanced security, resilience, and control

Publications (2)

Publication Number Publication Date
NZ782231A NZ782231A (en) 2023-08-25
NZ782231B2 true NZ782231B2 (en) 2023-11-28

Family

ID=

Similar Documents

Publication Publication Date Title
US11182247B2 (en) Encoding and storage node repairing method for minimum storage regenerating codes for distributed storage systems
US9992014B2 (en) Methods for cryptographic delegation and enforcement of dynamic access to stored data
US9203812B2 (en) Dispersed storage network with encrypted portion withholding and methods for use therewith
US9104691B2 (en) Securing data in a dispersed storage network using an encoding equation
Chen et al. Robust dynamic provable data possession
US9996413B2 (en) Ensuring data integrity on a dispersed storage grid
US20150169897A1 (en) Efficient and secure data storage utilizing a dispersed data storage system
US10447474B2 (en) Dispersed data storage system data decoding and decryption
US8601259B2 (en) Securing data in a dispersed storage network using security sentinel value
EP2340489B1 (en) Distributed storage and communication
US20100266120A1 (en) Dispersed data storage system data encryption and encoding
Chen et al. Robust dynamic remote data checking for public clouds
NZ782231B2 (en) Method and system for distributed data storage with enhanced security, resilience, and control
NZ782231A (en) Method and system for distributed data storage with enhanced security, resilience, and control
KR100561845B1 (en) Method for encrypting and decrypting data for multi-level access control in ad-hoc network
VS et al. A secure regenerating code‐based cloud storage with efficient integrity verification
EP4244749A1 (en) Method of ensuring confidentiality and integrity of stored data and metadata in an untrusted environment
Juels et al. Falcon codes: fast, authenticated lt codes (or: making rapid tornadoes unstoppable)
US10902144B2 (en) Method and apparatus for securing data
Nithisha et al. A study on effective mechanisms for secret sharing in Distributed Blockchain Systems
Vasu Techniques for efficiently ensuring data storage security in cloud computing
Bel et al. Inkpack
Bel et al. Inkpack: A Secure, Data-Exposure Resistant Storage System
Khemchyan et al. Data Sharing Based on Error-Correcting Codes
Paul et al. Design of a secure and fault tolerant environment for distributed storage