GB2444343A

GB2444343A - Encryption system for peer-to-peer networks in which data is divided into chunks and self-encryption is applied

Info

Publication number: GB2444343A
Application number: GB0709761A
Authority: GB
Inventors: David Irvine
Original assignee: Individual
Current assignee: Individual
Priority date: 2006-12-01
Filing date: 2007-05-22
Publication date: 2008-06-04
Anticipated expiration: 2027-05-22
Also published as: GB0624058D0; GB2446200A; GB0709761D0; GB2444343B

Abstract

A file is divided into chunks 2 and small data elements, such as bytes, are permuted between the chunks 3. Each chunk is hashed 4 to form a chunk hash, and each chunk hash is encrypted using one or more other chunk hashes 5 in a self-encryption process. Self-encryption involves obfuscation using algorithms seeded by elements preferably derived from the data itself so that the data holds the key to reversing the processes. The chunk hashes and the encrypted chunk hashes are constituents of a mapping function. The chunks are distributed across various nodes of a peer-to-peer network. The permutation of the chunks contributes towards the effective encryption of the file. The file can be recovered using the mapping function. Also disclosed is "duplicate removal" and "storing files".

Description

STATEMENT OF INVENTION:

8 An issue with today's encryption techniques is that a user's key, 9 biometric data or passphrase is used to encrypt every data element, thereby exposing the key on every data element encrypted. Another 11 issue is that eventually all encryption is broken given enough resources, 12 so it is therefore safe to assume that today's strong encryption methods 13 will not suffice in years to come. This implies that storing encrypted data 14 now, will not necessarily protect against that data being unencrypted through some discovered process in the future.

16 This present invention overcomes these issues by first obfuscating the 17 data, by splitting it into smaller elements, then swapping parts of that 18 data around in a manner to make every element useless on its own, 19 and preferably using known information from the preferably smaller elements or chunks as encryption data that will allow the other elements 21 to be encrypted. This allows data to be hidden and encrypted in such a 22 way, that any attacker would require to obtain all data elements and 23 know the manner in which they connect together and also then crack 24 the encryption used. Even if the data chunks were not encrypted and their encryption was broken, they are useless on their own.

BACKGROUND.

26 Selfencryption is only possible with combination of number of 27 elements. Described below is prior art for each element.

28 ENCRYPTION 29 W02005093582 discloses method of encryption where data is secured in the receiving node via private tag for anonymous network browsing.

31 However, other numerous encryption methods are also available such 32 as (I) implantation of Reed Solomon algorithm (W002052787), which 33 ensures data is coded in parabolic fashion for self-repairing and 34 storage1 (ii) storage involves incremental backup (W002052787), (ii) uses stenographic (US2006177094), (iv) use cipher keys (CN1620005), 36 encryption for non text (US20061 07048) and US20051 08240 discloses 37 user keys and randomly generated leaf node keys. The present 38 invention uses none of these methods of encryption and in particular 39 ensures all chunks are unique and do not point to another for security (an issue with Reed Solomon and N + K implementations of parabolic 41 coding) 43 SELF-ENcRYPTION 44 Attempts to moving towards attaining some limited aspects of self-encryption are demonstrated by: 46 (a) US2003053053625 discloses limitation of asymmetrical and 47 symmetrical encryption algorithms, and particularly not requiring 48 generation of a key stream from symmetric keys, nor requiring any time 49 synchronizing, with minimal computational complexity and capable of operating at high speed. A serial data stream to be securely transmitted 51 is first demultiplexed into a plurality N of encryptor input data stream.

52 The input data slices are created which have a cascade of stages, 53 include mapping & delay functions to generate output slices. These are 54 transmitted though a transmission channel. Decryptor applies inverse step of cascade of stages, equalizing delay function and mapping to 56 generate output data slices. The output data streams are multiplexed.

57 The encryptor and decryptor require no synchronizing or timing and 58 operate in simple stream fashion. N:N mapping does not require 59 expensive arithmetic and implemented in table lookup. This provides robust security and efficiency. A significant difference between this 61 approach and prior cipher method is that the session key is used to 62 derive processing parameters (tables and delays) of the encryptor and 63 decryptor in advance of data transmission. Instead of being used to 64 generate a key stream at real-time rates. Algorithm for generating parameters from a session key is disclosed. This is a data 66 communications network and not related to current invention.

67 (b) US2002184485 addresses secure communication, by encryption of 68 message (SSDO-self signing document objects), such that only known 69 recipient in possession of a secret key can read the message and verification of message, such that text and origin of message can be 71 verified. Both capabilities are built into message that can be transmitted 72 over internet and decrypted or verified by computer implementing a 73 document representation language that supports dynamic content e.g. 74 any standard web browser, such that elaborate procedures to ensure transmitting and receiving computers have same software are no longer 76 necessary. Encrypted message or one encoded for verification can 77 carry within itself all information needed to specify the algorithm needed 78 for decryption.

Summary of Invention

79 The main embodiments of this invention are as follows: A system of self encryption which has the functional elements of: 81 1. Duplicate Removal 82 2. Storing Files 83 3. Chunking 84 4. Encryption / Decryption 85... with the additionally linked functional elements of: 86 1. Identify Chunks 87 2. Self Healing 88 3. Storage and Retrieval 89 4. Security Availability 5. Provision of Key Pairs 91 A system of self-encryption of data in a distributed and peer to peer 92 network 93 A product for self-encryption of data in a distributed and peer to peer 94 network A system to provide self-encryption in a distributed network which is 96 made of inter linkage all or some of the following elements; 97 a. encryption / decryption 98 b. chunking 99 c. duplicate removal d. storing files 101 A system to provide self-encryption in a distributed network which is 102 made of inter linkage all or some of the following elements and sub- 103 elements; 104 a. encryption / decryption i. key pair 106 ii. security 107 b chunking 108 i. identify chunking 109 c. duplicate removal i. identify chunking 111 ii. storage & retrieval 112 iii. self healing 113 d. storing fUes 114 i. identify chunking ii. storage & retrieval 116 iii. self healing 117 A product for self-encryption in a distributed network which is made of 118 inter linkage all or some of the following elements; 119 a, encryption / decryption b. chunking 121 c. duplicate removal 122 d. storing files 123 A product for self-encryption in a distributed network which is made of 124 inter linkage all or some of the following elements and sub-elements; a encryption / decryption 126 i. key pair 127 ii. security 128 b. chunkirig (p - 129 i. identify chunking c. duplicate removal 131 i. identify chunking 132 ii. storage & retrieval 133 iii self healing 134 d. storing files i. identify chunking 136 ii. storage & retrieval 137 iii. self healing 138 A method of system and product for self-encryption of data in a 139 distributed and peer to peer network A method of above of securely protecting data in a distributed network, 141 suitable for a self repairing process by chunking the data into many 142 pieces.

143 A method of above where data privacy by byte or bit exchange and 144 encryption is based on content derived from the data itself.

A method of above where data reconstitution capability is provided only 146 for individuals who know of and/or have the original data elements.

147 A method of maximising disk space in a worldwide network by aiding 148 the removal of duplicate files, as each data element will always produce 149 the exact same chunks and names regardless of the actual file name itself.

151 A method of data encryption using only calculable elements from the file 152 contents and not user keys or user passwords.

153 A method of above where the actual file is first passed though a content 154 swapping (such as byte swapping)algorithm to completely dilute the contents across the data element(s), thereby rendering each chunk 156 useless even if the encryption key is known.

DESCRIPTION

Detailed Description:

157 (References to I Os used in descriptions of the system's functionality) 158 MID -this is the base ID and is mainly used to store and forget files.

159 Each of these operations will require a signed request. Restoring may simply require a request with an ID attached.

161 PMID -This is the proxy mid which is used to manage the receiving of 162 instructions to the node from any network node such as get/ put / forget 163 etc This is a key pair which is stored on the node -if stolen the key pair 164 can be regenerated simply disabling the thief's stolen PMID -although there's not much can be done with a PMID key pair.

166 CID -Chunk Identifier, this is simply the chunkid.KID message on the 167 net.

168 TMID -This is today's ID a one time ID as opposed to a one time 169 password. This is to further disguise users and also ensure that their MID stays as secret as possible.

171 MPID -The maidsafe.net public ID. This is the IDto which users can 172 add their own name and actual data if required. This is the ID for 173 messenger, sharing, non anonymous voting and any other method that 174 requires we know the user.

MAID -this is basically the hash of and actual public key of the MID.

176 this ID is used to identify the user actions such as put / forget / get on 177 the maidsafe.net network. This allows a distributed PKI infrastructure to 178 exist and be automatically checked.

179 KID -Kademlia ID this can be randomly generated or derived from known and preferably anonymous information such as an anonymous 181 public key hash as with the MAID.. In this case we use kademlia as the 182 example overlay network although this can be almost any network 183 environment at all.

184 MSID -maidsafe.net Share ID, an ID and key pair specifically created for each share to allow users to interact with shares using a unique key 186 not related to their MID which should always be anonymous and 187 separate Linked elements for Self Encryption (Figure 1 -PT2) 188 The Self Encryption invention consists of 4 key functional elements, with 189 a further 5 functional elements being linked with The key functional elements are 191 P5-Duplicate Removal 192 P6 -Storing Files 193 P7 -Chunking 194 P8 -Encryption I Decryption The linked functional elements are: 196 P9 -Identify Chunks 197 P2 -Self Healing 198 P4 -Storage and Retrieval 199 P3 -Security Availability P13 -Provision of Key Pairs 202 The self-encryption (PT2) itself is made up from linkage of elements, 203 storing file (P6), duplicate removal (P5), chunking (P7) and encryption I 204 decryption (P8) which allows a self-encryption process to provide security to 205 and global duplicate data removal. In addition, storing file element (P6) is 206 preferably dependent upon sub-elements storage and retrieval (P4) and 207 sub-element identify chunks (P9) and generate sub-element self-healing 208 (P2), duplicate removal element (P5) is preferably dependent on sub- 209 element identify chunks (P9), chunking element (P7) generate sub- 210 element identify chunks (P9) and encryption / decryption element (P8) 211 can be provided by sub-element provision of keys (P 13) to ensure validity 212 of generating or requesting nodes anonymous identity (e.g. we don't 213 know who it is but we know it was the node that put the chunk there) 214 thereby ensuring security availability (P3).

Chunking (Figure 1-P7) 215 According to a related aspect of this invention, files are split 216 preferably using an algorithm to work out the chunk size into several 217 component parts. The size of the parts is preferably worked out from 218 known information about the file as a whole, preferably the hash of the 219 complete file. This information is run through an algorithm such as 220 adding together the first x bits of the known information and using 221 modulo division to give a chunk size that allows the file to preferably 222 split into at least three parts.

223 Preferably known information from each chunk is used as an encryption 224 key. This is preferably done by taking a hash of each chunk and using 225 this as the input to an encryption algorithm to encrypt another chunk in 226 the file. Preferably this is a symmetrical algorithm such as AES256.

227 Preferably this key is input into a password creating algorithm such as 228 pbkdf and an initial vector and key calculated from that. Preferably the 229 iteration count for the pbkdf is calculated from another piece of known 230 information, preferably the sum of bits of another chunk or similar.

231 Preferably each initial chunk hash and the final hash after encryption 232 are stored somewhere for later decryption.

Self Enctypting Files (Figure 2a/b) 233 1. Take a content hash of a file or data element 234 2. Chunk a file with preferably a random calculable size i.e. based on an 235 algorithm of the content hash (to allow recovery of file). Also obfuscate 236 thefilesuchasin3 237 3. Obfuscate the chunks to ensure safety even if encryption is eventually 238 broken (as with all encryption if given enough processing power and time) 239 a. chunk 1 byte 1 swapped with bytel of chunk 2 240 b. chunk 2 byte 2 swapped with byte I chunk 3 241 c. chunk 3 byte 2 swapped with byte 2 of chunk 1 242 d. This repeats until all bytes swapped and then repeats the same 243 number of times as there are chunks with each iteration making next 244 chunk first one 245 e. -i.e. second time round chunk 2 is starting position 246 4. Take hash of each chunk and rename chunk with its hash.

247 5. Take h2 and first x bytes of h3 (6 in our example case) and either use 248 modulo division or similar to get a random number between 2 fixed 249 parameter (in our case 1000) to get a variable number. Use the above 250 random number and h2 as the encryption key to encrypt hi or use h2 and 251 the random number as inputs to another algorithm (pdbfk2 in our case) to 252 create a key and iv.(initialisation vector) 2.

253 6. This process may be repeated multiple times to dilute any keys 254 throughout a series of chunks.

255 7. Chunk name i.e. hi (unencrypted) and hic (and likewise for each chunk) 256 written to a location for later recovery of the data. Added to this we can 257 simply update such a location with new chunks if a file has been altered, 258 thereby creating a revision control system where each file can be rebuilt 259 to any previous state.

260 8. The existence of the chunk will be checked on the net to ensure it is not 261 already backed up. All chunks may be checked at this time.

262 9. If a chunk exists all chunks must be checked for existence.

263 10. Thechunk is saved 264 11. The file is marked as backed up.

265 12. If a collision is detected the process is redone altering the original size 266 algorithm (2) to create a new chunk set, each system will be aware of this 267 technique and will do the exact same process till a series of chunks do 268 not collide. There will be a back off period here to ensure the chunks are 269 not completed due to the fact another system is backing up the same file.

270 The original chunk set will be checked frequently in case there are false 271 chunks or ones that have been forgotten. if the original names become 272 available the file is reworked using these parameters.

Duplicate Removal (Figure 1 -P5) 273 According to a related aspect of this invention, data chunked and 274 ready for storing can be stored on a distributed network but a search 275 should preferably be carried out for the existence of all associated 276 chunks created. Preferably the locations of the chunks have the same 277 ranking (From earlier ranking system) as user or better, otherwise the 278 existing chunks on the net are promoted to a location of equivalent rank 279 at least. If all chunks exist then the file is considered as already backed 280 up. If less than all chunks exist then this will preferably be considered 281 as a collision (after a time period) and the file will be re chunked using 282 the secondary algorithms (preferably just adjusted file sizes). This 283 allows duplicate files on any 2 or more machines to be only backed up 284 once, although through perpetual data several copies will exist of each 285 file, this is limited to an amount that will maintain perpetual data.

Enciypt -Deci'ypt (Figure 1 -PS) 286 According to a related aspect of this invention, the actual encrypting 287 and decrypting is carried out via knowledge of the file's content and this 288 is somehow maintained (see next). Keys will be generated and 289 preferably stored for decrypting. Actually encrypting the file will 290 preferably include a compression process and further obfuscation 291 methods. Preferably the chunk will be stored with a known hash 292 preferably based on the contents of that chunk.

293 Decrypting the file will preferably require the collation of all chunks and 294 rebuilding of the file itself. The file may preferably have its content 295 mixed up by an obfuscation technique rendering each chunk useless on 296 its own 297 Preferably every file will go through a process of byte (or preferably bit) 298 swapping between its chunks to ensure the original file is rendered 299 useless without all chunks.

300 This process will preferably involve running an algorithm which 301 preferably takes the chunk size and then distributes the bytes in a g4 302 pseudo random manner preferably taking the number of chunks and 303 using this as an iteration count for the process. This will preferably 304 protect data even in event of somebody getting hold of the encryption 305 keys -as the chunks data is rendered useless even if transmitted in the 306 open without encryption.

307 This defends against somebody copying all data and storing for many 308 years until decryption of today's algorithms is possible, although this is 309 many years away.

310 This also defends against somebody; instead of attempting to decrypt a 311 chunk by creating the enormous amount of keys possible, (in the region 312 of 2A54) rather instead creating the keys and presenting chunks to all 313 keys -if this were possible (which is unlikely) a chunk would decrypt.

314 The process defined here makes this attempt useless.

315 All data will now be considered to be diluted throughout the original 316 chunks and preferably additions to this algorithm will only strengthen 317 the process.

Security (Figure 1 -P3) 318 According to a related aspect of this invention, each file is split into 319 small chunks and encrypted to provide security for the data. Only the 320 person or the group, to whom the overall data belongs, will know the 321 location of the other related but dissimilar chunks of data.

322 Preferably, each of the above chunks does not contain location 323 information for any other dissimilar chunks; which provides for security 324 of data content, a basis for integrity checking and redundancy.

325 Preferably, the method further comprises the step of only allowing the 326 person (or group) to whom the data belongs to have access to it, 327 preferably via a shared encryption technique which allows persistence 328 of data.

329 Preferably, the checking of data or chunks of data between machines is 330 carried out via any presence type protocol such as a distributed hash

331 table network.

332 Preferably, on the occasion when all data chunks have been relocated, 333 i.e. the user has not logged on for a while, a redirection record is 334 created and stored in the super node network, (a three copy process - 335 similar to data) therefore when a user requests a check, the redirection 336 record is given to the user to update their database, which provides 337 efficiency that in turn allows data resilience in cases where network 338 churn is a problem as in peer to peer or distributed networks. This 339 system message can be preferably passed via the messenger system 340 described herein.

341 Preferably the system may simply allow a user to search for his chunks 342 and through a challenge response mechanism, locate and authenticate 343 himself to have authority to getiforget this chunk.

344 Further users can decide on various modes of operation preferably 345 such as maintain a local copy of all files on their local machine, 346 unencrypted or chunked or chunk and encrypt even local files to secure 347 machine (preferably referred to as off line mode operation) or indeed 348 users may decide to remove all local data and rely completely on 349 preferably maidsafe.net or similar system to secure their data.

Claims

350 1. A system to provide self-encryption in a distributed network which is 351 made of combination and inter linkage of all or some of the following 352 elements; 353 a. encryption I decryption 354 b. chunking 355 c. duplicate removal 356 d. storing files 357
2. A system of claim 1 to provide self-encryption in a distributed network 358 which is made of combination and inter linkage of all or some of the 359 following elements and sub-elements; 360 a. encryption / decryption 361 I. key pair 362 ii. security 363 b. chunking 364 i. identify chunking 365 c. duplicate removal 366 i. identify chunking 367 ii. storage & retrieval 368 iii. self healing 369 d. storing files 370 i. identify chunking 371 ii. storage & retrieval 372 iii. self healing 373
3. A product for self-encryption in a distributed network which is made of 374 combination and inter linkage of all or some of the following elements; 375 a. encryption I decryption 376 b. chunking 377 c. duplicate removal 378 d storing files 379
4. A product for self-encryption in a distributed network which is made of 380 combination and inter linkage of all or some of the following elements and 381 sub-elements; 382 a encryption I decryption 383 i. key pair 384 ii. security 385 b. chunking 386 i. identify churiking 387 c. duplicate removal 388 I. identify chunking 389 ii. storage & retrieval 390 iii. self healing 391 d. storing files 392 i. identify chunking 393 ii. storage & retrieval 394 iii. self healing 395
5. A method of claim 1-4 where it is to identify data elements using a data 396 map with only a sequence of content hashes for each chunk of data 397 before and after encryption; 398
6. A method of claims 1-5 storing and retrieving these maps on an insecure 399 network; 400
7. A method of claim 5 where each, new iteration of a data element is 401 appended to the data map to create a strong revision control system; 402
8. A method of claim 5 where data elements are obfuscated by encryption 403 or other obfuscation technique, or similar, can be reconstructed in 404 conjunction with the data map; 405
9. A method of claim 5 where the maps can be stored in private or public 406 locations and/or biometrically accessed: 407
10. A system of claims 1-2 which allows data to have multiple locations, 408 revisions and encryption or other obfuscation techniques and for the 409 pointer to the data to be a very small file containing the basic information 410 to reconstitute a complete data element at any time from any location on 411 the network; 412
11. A system of claims 1-2 which allows the identification of which chunks to 413 make up which files; 414
12. A system of claims 1-2 which allows data maps which preferably become 415 discreet data chunks on the network, just like any other associated data 416 element and are therefore undetectable as data maps;