CN116303329A - Distributed storage method based on block chain - Google Patents
Distributed storage method based on block chain Download PDFInfo
- Publication number
- CN116303329A CN116303329A CN202310593970.6A CN202310593970A CN116303329A CN 116303329 A CN116303329 A CN 116303329A CN 202310593970 A CN202310593970 A CN 202310593970A CN 116303329 A CN116303329 A CN 116303329A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- blockchain
- node
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 239000012634 fragment Substances 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 7
- 238000013467 fragmentation Methods 0.000 claims description 5
- 238000006062 fragmentation reaction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed storage method based on a blockchain, which is characterized in that after an account obtains storage authority, a file storage task is initiated, and the method comprises the following steps: s1, the front end segments the file to obtain content identifiers of the segments of each byte
The method comprises the steps of carrying out a first treatment on the surface of the S2, slicing all bytes into content identifiers
Constructed as a Merkle tree (Merkle tree) to obtain the root of the tree
This is
I.e. the whole document
The method comprises the steps of carrying out a first treatment on the surface of the S3, acquiring on-line storage node information, and hashing node addresses and files
Performing exclusive OR calculation to obtain the distance between each node and the file; s4, selecting one or more nearest nodes for storage according to the distance. The invention combines the distributed storage of files and the on-chain storage of files, and the content identifier of the files
And (e.g. hash) cannot be tampered after being linked, and whether the traceable file is tampered, the information of an uploader and the like.
Description
Technical Field
The invention relates to the field of distributed storage and blockchain, in particular to a blockchain-based distributed storage method.
Background
The distributed storage system disperses and stores data on a plurality of independent devices. The traditional network storage system adopts a centralized storage server to store all data, and the storage server becomes a bottleneck of system performance, is also a focus of reliability and safety, and cannot meet the requirements of large-scale storage application. The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position servers to position the storage information, thereby improving the reliability, availability and access efficiency of the system and being easy to expand.
Traditional distributed systems such as HDFS use a central server route that keeps track of which machines the file is stored on and will not be available once the central server is damaged.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The present invention is directed to solving the above-mentioned problems, and provides a distributed storage method based on a blockchain.
In order to achieve the above object, the technical scheme of the present invention is as follows:
the distributed storage method based on the block chain, after the account obtains the storage authority, the file storage task is initiated, and the method comprises the following steps: s1, the front end segments the file to obtain content identifiers of the segments of each byte
The method comprises the steps of carrying out a first treatment on the surface of the S2, content identifier of slicing all bytes +.>
Constructed as a Merkle tree, the whole file is obtained>
The size of the storage space required; s3, acquiring online storage node information, hashing node addresses and adding +.>
Performing exclusive OR calculation to obtain the distance between each node and the file; s4, selecting one or more nearest nodes for storage according to the distance.
Further, in step S4, the specific steps for storing are as follows: s41, uploading the file to one or more nodes closest to the file and storing the file in a temporary folder, S42, assembling related information stored in the file into a blockchain transaction, signing the blockchain transaction, sending the transaction, and then asynchronously returning transaction hash and file uploading state information, S43, calling contracts by the blockchain, and storing the file.
Further, the account needs to acquire storage rights through an in-chain transaction.
Further, when the storage node stores the file, the merkel tree of the file is stored in a cache server database local to the current storage node.
In addition, the invention also provides a distributed storage system based on the block chain, which comprisesThe device comprises a slicing module, a merkel tree construction module, a distance calculation module and a storage module; the slicing module slices the file to obtain each byte slice
The method comprises the steps of carrying out a first treatment on the surface of the The merkel tree construction module recursively slices the bytes obtained by the slicing module>
A merkel tree structured as a file; the distance calculation module hashes the node address and the +.>
And performing exclusive OR calculation to obtain the distance between each node and the file.
The storage module comprises a fragmentation storage module for storing file byte fragments and a local cache server database module for storing file merkel trees.
In addition, the invention also provides a distributed storage device based on the block chain, which comprises: the system comprises a memory, a processor and a distributed storage program based on a block chain, wherein the distributed storage program based on the block chain is stored on the memory and can run on the processor, and is configured with a distributed storage method based on the block chain.
The invention also provides a storage medium, on which a blockchain-based distributed storage program is stored, which when executed, implements a blockchain-based distributed storage method.
By adopting the technical scheme, the invention has the following advantages:
1. file distributed storage and on-chain combination, content identifier of file
And (e.g. hash) cannot be tampered after being linked, and whether the traceable file is tampered, the information of an uploader and the like.
2. For files when stored
The hash of the storage node address is exclusive-or calculated and stored in the nearest nodes according to the distance sorting, so that the server on which the file is located is not needed to be known when the file is searched, the nearest nodes can be generally searched, the storage position of the file is not needed to be recorded by the server, and the file is further decentralized.
3. Conventional rights systems are recorded in an under-chain database, centralized, and tamperable. The authority system is recorded on the chain, and the transaction needs to be initiated on the chain, and the packaged transaction blocks and the consensus can not pass through the system.
4. The conventional file uploading does not need to interact with the blockchain, and the file uploading can be carried out only by initiating a transaction on the chain, packaging the transaction out of a block and consensus.
5. The file storage structure adopts the data structure of the merkle tree, and can provide the merkle proof for verifying whether the file block is tampered.
Drawings
FIG. 1 is a schematic diagram of the present invention's Meeker tree structure.
FIG. 2 is a schematic diagram of the structure and flow of the present invention.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Example 1: the invention discloses a distributed storage method based on a blockchain, which comprises a participation node, a consensus node and an creation node. The generation node is the first node of the initial chain and is also a consensus node; the consensus node can perform consensus on the blocks and can initiate transaction; the participating nodes may initiate transactions.
When the chain is started, relevant parameters of distributed storage need to be configured, such as:
parameter DFS (whether node opens distributed storage) =true;
parameter GinHTTPPort (distributed storage gin port) =3004;
the parameter ChunkSize (file fragment size, default 256kB, i.e., 26262309B) =2626144;
the parameter nodaacount (number of leaf nodes, default 100) =100.
As shown in fig. 2, the account has storage authority to store the file, so the authority needs to be applied first. The application of rights is common general knowledge and is only briefly described here. In this embodiment, the consensus node account may apply for storage rights for other accounts. And sending the signed transaction, wherein the parameters comprise the applied account address and the storage space allocated for the account address, calling a distributed storage authority preset contract to update the storage authority, and uploading the block consensus through the post-transaction.
After the account obtains the storage rights, a storage transaction may be initiated.
The file storage includes the steps of:
s1, storing a file into a temporary folder by the front end, and slicing the file to obtain content identifiers of slicing of each byte
；
S2, slicing all bytes into content identifiers
Constructed as a Merkle tree, the whole file is obtained>
The size of the storage space required;
s3, acquiring on-line storage node information, and hashing node addresses and files
Performing exclusive OR calculation to obtain the distance between each node and the file;
s4, selecting one or more nearest nodes to store redundantly according to the distance.
The specific implementation method of the step S1 is as follows: reading the file according to the fixed size, slicing to obtain byte slices, and performing hash operation on each obtained byte slice (for example, first sha256 calculation and then passingbase58 code) to obtain each byte slice
。
In step S2, the bytes are sliced by recursion
Constructed as a merkel tree, i.e. each byte slice as a leaf node, byte slice +.>
I.e. leaf node +.>
Generating upper nodes on the upper layer of all leaf nodes, wherein +.>
(i.e.)>
) Is through +.>
And performing hash operation. After construction the whole file can be obtained +.>
(i.e., merkle root of Merkle tree), the required storage size.
The fixed size typically reads one byte slice in 256KB size, which in turn generates leaf nodes.
As shown in FIG. 1, the file of this embodiment is assumed to be divided into 199 byte slices, the number of leaf nodes is set to be at most 100 per layer, and after the number of leaf nodes in each layer reaches 100, an upper node is created in the upper layer, where the upper node is
(i.e.)>
) By +.about.all leaf nodes of the lower layer>
And performing hash operation to obtain the product. And constructing from bottom to top in this way, and finally generating root nodes of the Meacker tree. The upper node has Links which record the cid of all leaf nodes to which the upper node belongs in sequence and point to all leaf nodes hung on the upper node in sequence.
The data structure of the upper node may include: storing a file Name ", such as" test. Txt ";
links ", such as [" QmeNp2Fc9 cKFBWVrmmwnn 6vNUngjmZgBwnz93kwJVLLryvN ","9kbWQfR3Kwrh5sv5SiwjmXna6DBJBf23PfQmUiPF2unS "," EBRHPpteK4uwe3874Wc59u7 yKAyGBQHEJAsRegmofpt "];
content identifier "cid" such as "8HPAfznRqxxAvYe3wQ2Pxfh7caKfCsY94 olalj 96epZc1".
After the required storage space is obtained in step S2, the storage space of the storage account can be checked by calling a distributed storage preset contract, and if the storage space is insufficient, the subsequent steps are not required to be triggered, so that resources are not wasted. Of course, this step may be performed after the actual storage.
In step S4, the specific method for performing redundancy storage is as follows: s41, uploading the file to one or more nodes closest to the file, and storing the file in a temporary folder of the node;
s42, storing the related information of the file, such as the file
The size of the required storage space, the temporary storage file path and address information of one or more nodes stored in a redundant mode are assembled into a blockchain transaction, signature is carried out, and transaction hash and file uploading state information are asynchronously returned after the transaction is sent;
s43, calling a distributed storage file uploading preset contract by a blockchain, calling file uploading logic in a predictor, wherein parameters comprise node addresses and temporary file paths to be stored, then triggering the file uploading logic in the predictor to judge whether the nodes are nodes to be stored or not, if yes, finding storage files in a temporary folder, storing the files in byte fragments after the files are fragmented (the fragments are consistent with the fragmentation method in the step S1), storing the merkel tree of the files in a cache server database local to the current storage node, and deleting the temporary files.
Considering the actual storage scene, the embodiment is mainly used for storing files of enterprises, nodes are fewer and limited, the nodes are not frequently added and withdrawn, 2 redundant parts of all files are set, 1 part of uploading node is stored, and 3 parts of uploading node is used. In this embodiment, the uploading node will store the Merkle root in the node local cache server database; the first two bits of each fragment are used for fetching the cid to create a folder, the folder is stored in a local file system, and if the block is found to exist in the file system during storage, the file is not stored repeatedly.
Step S42 is to asynchronously return the result, and the consensus and storage process is performed after the result is returned, so that the speed of the file uploading interface is improved, the large file storage speed is prevented from being slow, the waiting time of the user is long, and high concurrency is supported. The user can be based on the file
And querying the file to store the result.
When the file needs to be downloaded, the file is firstly according to the file
Inquiring Merkel tree information in a local cache server database related to the file, judging whether the node has the file, if so, finding each fragment of the file through merkle tree information, and assembling and returning; and if not, the adjacent storage nodes are downloaded in sequence according to the distance of exclusive OR calculation with the file until the adjacent nodes have the file returned or all the nodes are queried.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the object of the present invention, which is not limited herein.
Example 2: in addition, the embodiment of the invention also provides a distributed storage system based on the blockchain, which is shown in fig. 2 and comprises a slicing module, a merkel tree construction module, a distance calculation module and a storage module.
The slicing module slices the file to obtain each byte slice
。
The merkel tree construction module recursively fragments the bytes obtained by the fragmentation module
Structured as a merkel tree of files.
The distance calculation module hashes the node address and files
And performing exclusive OR calculation to obtain the distance between each node and the file.
The storage module comprises a fragmentation storage module for storing file byte fragments and a local cache server database module for storing a merkel tree.
In addition, the present embodiment is merely a basic description of the blockchain-based distributed storage system of the present invention, and technical details that are not described in detail in this embodiment may refer to the methods provided in any embodiment of the present invention, which are not described herein again.
Example 3: it will be apparent to those skilled in the art that the system and method of the above embodiments may be implemented by software plus a necessary general purpose hardware platform, but may also be implemented by hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a node packaging device, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The present invention therefore also provides a blockchain-based distributed storage device comprising: the system comprises a memory, a processor and a distributed storage program based on a block chain, wherein the distributed storage program based on the block chain is stored on the memory and can run on the processor, and is configured with a distributed storage method based on the block chain.
In addition, the invention also provides a storage medium, wherein the storage medium is stored with a distributed storage program based on the blockchain.
Since, in a real situation, when a device or a program is deployed, the execution of a certain program may be to perform all steps, or may be to perform only a certain step, and implement all steps by cooperation of a plurality of programs,
therefore, the blockchain-based distributed storage program, when executed, implements all or some of the processes of the blockchain-based distributed storage method.
The foregoing description of the preferred embodiments of the present invention should not be taken as limiting the scope of the invention, but rather should be understood to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the following description and drawings.
Claims (8)
1. The distributed storage method based on the block chain is characterized in that after an account obtains storage authority, a file storage task is initiated, and the method comprises the following steps: s1, the front end segments the file to obtain content identifiers of the segments of each byte
；
S2, slicing all bytes into content identifiers
Constructed as a merkel tree and is,get +.>
The size of the storage space required;
s3, acquiring on-line storage node information, and hashing node addresses and files
Performing exclusive OR calculation to obtain the distance between each node and the file;
s4, selecting one or more nearest nodes for storage according to the distance.
2. The blockchain-based distributed storage method of claim 1, wherein in step S4, the specific steps of storing are as follows: s41, uploading the file to one or more nodes closest to the file, and storing the file in a temporary folder of the node;
s42, storing the related information of the file
Assembling a blockchain transaction, signing, sending the transaction, and then asynchronously returning transaction hash and file uploading state information, wherein the related information comprises +_ of the file>
;
S43, invoking contracts by the block chain to store the file.
3. The blockchain-based distributed storage method of claim 1, wherein the account requires access to storage rights through an on-chain transaction.
4. The blockchain-based distributed storage method of claim 1, wherein when the storage node stores the file, the merkel tree of the file is stored in a cache server database local to the current storage node.
5. The distributed storage system based on the block chain is characterized by comprising a slicing module, a merkel tree construction module, a distance calculation module and a storage module;
the slicing module slices the file to obtain each byte slice
；
The merkel tree construction module recursively fragments the bytes obtained by the fragmentation module
A merkel tree structured as a file;
the distance calculation module hashes the node address and files
And performing exclusive OR calculation to obtain the distance between each node and the file.
6. The blockchain-based distributed storage system of claim 5, wherein the storage module includes a sharded storage module for storing file byte shards and a local cache server database module for storing file merkel trees.
7. A blockchain-based distributed storage device comprising a memory, a processor, and a blockchain-based distributed storage program stored on the memory and executable on the processor, the blockchain-based distributed storage program configured to implement the blockchain-based distributed storage method of any of claims 1-4.
8. A storage medium having stored thereon a blockchain-based distributed storage program that, when executed, implements the blockchain-based distributed storage method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310593970.6A CN116303329A (en) | 2023-05-25 | 2023-05-25 | Distributed storage method based on block chain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310593970.6A CN116303329A (en) | 2023-05-25 | 2023-05-25 | Distributed storage method based on block chain |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116303329A true CN116303329A (en) | 2023-06-23 |
Family
ID=86785528
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310593970.6A Pending CN116303329A (en) | 2023-05-25 | 2023-05-25 | Distributed storage method based on block chain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116303329A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256663A (en) * | 2020-10-30 | 2021-01-22 | 深圳壹账通智能科技有限公司 | Block chain based distributed file storage method and system |
| US20210248271A1 (en) * | 2020-02-12 | 2021-08-12 | International Business Machines Corporation | Document verification |
| CN113572618A (en) * | 2021-08-10 | 2021-10-29 | 东北大学 | Fabric and IPFS combined decentralized storage system and data storage method thereof |
| KR20220066466A (en) * | 2020-11-16 | 2022-05-24 | 두나무 주식회사 | Method and apparatus for managing document based on blockchain network |
| CN115204873A (en) * | 2022-07-07 | 2022-10-18 | 北京航空航天大学云南创新研究院 | Block chain-based open big data transaction matching method and system |
| CN115348275A (en) * | 2022-07-11 | 2022-11-15 | 一诺(深圳)离岸云数据科技有限公司 | Offshore data distributed storage system based on block chain quick addressing technology |
-
2023
- 2023-05-25 CN CN202310593970.6A patent/CN116303329A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210248271A1 (en) * | 2020-02-12 | 2021-08-12 | International Business Machines Corporation | Document verification |
| CN112256663A (en) * | 2020-10-30 | 2021-01-22 | 深圳壹账通智能科技有限公司 | Block chain based distributed file storage method and system |
| KR20220066466A (en) * | 2020-11-16 | 2022-05-24 | 두나무 주식회사 | Method and apparatus for managing document based on blockchain network |
| CN113572618A (en) * | 2021-08-10 | 2021-10-29 | 东北大学 | Fabric and IPFS combined decentralized storage system and data storage method thereof |
| CN115204873A (en) * | 2022-07-07 | 2022-10-18 | 北京航空航天大学云南创新研究院 | Block chain-based open big data transaction matching method and system |
| CN115348275A (en) * | 2022-07-11 | 2022-11-15 | 一诺(深圳)离岸云数据科技有限公司 | Offshore data distributed storage system based on block chain quick addressing technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11556516B2 (en) | Distributed blockchain data storage under account model | |
| EP3837652B1 (en) | Distributed blockchain data storage under account model | |
| US10862672B2 (en) | Witness blocks in blockchain applications | |
| US11836112B2 (en) | Path resolver for client access to distributed file systems | |
| US20210303633A1 (en) | Shard hashing | |
| EP3769230B1 (en) | Taking snapshots of blockchain data | |
| US11194792B2 (en) | Taking snapshots of blockchain data | |
| CN116204575A (en) | Method, device, equipment and computer storage medium for importing data into database | |
| CN111432009B (en) | Method and device for synchronizing block chain data, electronic equipment and storage medium | |
| CN116578746A (en) | Object de-duplication method and device | |
| CN116303329A (en) | Distributed storage method based on block chain | |
| CN113590703B (en) | ES data importing method and device, electronic equipment and readable storage medium | |
| CN110321326B (en) | Block chain file fragment retrieval method | |
| CN113986835A (en) | Management method, device, equipment and storage medium for FastDFS distributed files | |
| US11170000B2 (en) | Parallel map and reduce on hash chains | |
| CN113313540B (en) | Contract generation method and device, electronic equipment and readable storage medium | |
| CN115705344A (en) | Event processing method, electronic device and storage medium | |
| CN115203159B (en) | Data storage method, device, computer equipment and storage medium | |
| CN115840746A (en) | Data sub-table processing method and device, computer equipment and storage medium | |
| CN116719859A (en) | Data mobilizing method and device, storage medium and electronic device | |
| CN115470293A (en) | Data processing method, device, equipment and storage medium | |
| CN117435560A (en) | Data query method, device, electronic equipment and readable storage medium | |
| CN116680729A (en) | Data processing method and device based on intelligent contract and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230623 |
|
| RJ01 | Rejection of invention patent application after publication |