CN113986143A - Block chain log storage-oriented high-reliability low-overhead data storage method - Google Patents
Block chain log storage-oriented high-reliability low-overhead data storage method Download PDFInfo
- Publication number
- CN113986143A CN113986143A CN202111333973.3A CN202111333973A CN113986143A CN 113986143 A CN113986143 A CN 113986143A CN 202111333973 A CN202111333973 A CN 202111333973A CN 113986143 A CN113986143 A CN 113986143A
- Authority
- CN
- China
- Prior art keywords
- log
- storage
- client
- node
- log file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/382—Payment protocols; Details thereof insuring higher security of transaction
- G06Q20/3825—Use of electronic signatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q20/401—Transaction verification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Computer Interaction (AREA)
- Computer Security & Cryptography (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-reliability low-overhead data storage system and a method for block chain log storage, wherein the method comprises the following steps: dividing a log file to be stored into two storage stages, wherein the two storage stages comprise: a preparation phase and a submission phase; and storing the log file into the block chain log storage system in stages. The invention completes the storage of the log file by the submission of the log file into a preparation stage and a submission stage, can effectively reduce the storage overhead and expand the storage capacity of the system by multiplexing the physical node where the block chain system is located as the down-chain storage node under the condition of not increasing additional machines, and simultaneously avoids the deletion or the tampering of the data by the Byzantine node by combining the redundancy capability provided by the erasure coding.
Description
Technical Field
The invention relates to the technical field of distributed storage, in particular to a high-reliability low-overhead data storage system and method for block chain log storage.
Background
The blockchain is a technology which combines methods such as distributed consensus, encryption and time stamping and realizes point-to-point transaction, coordination and cooperation without depending on any third-party centralized mechanism. However, the blockchain has a problem of excessive storage overhead, and becomes a bottleneck of landing of blockchain applications. For the problem of too large storage overhead in a block chain, existing solutions can be divided into on-chain storage and off-chain storage. The data are stored in the block chain through on-chain storage, an additional under-chain storage system is not needed, and each node only needs to store corresponding data according to a preset rule, so that the storage cost is reduced, and the on-chain storage can be divided into a collaborative storage mode and a light node mode. The data in the block is transferred to a storage system outside the block chain by the down-chain storage, at this time, the block chain only stores non-data information such as pointers pointing to the data, and the storage space occupied by the non-data information is small, so that the problem of expandability of the block chain storage can be solved. Original data is stored in a non-block chain system, and a unique identifier of the data is generated according to a certain rule and returned to the block chain system; when the complete data is accessed, the original data is found in the non-blockchain system through the unique identification of the data.
The prior art discloses a method, a system and a device for data access, wherein during data storage, a data storage instruction is determined first, then a block chain and a key pair corresponding to an identity are determined according to the identity carried in the data storage instruction, finally data to be stored are stored in the block chain according to the key pair, during data query, a data query instruction is determined first, then the block chain and a private key corresponding to the identity are determined according to the identity corresponding to the data query instruction, and finally the data in the block chain are decrypted and queried according to the private key. Therefore, by the method provided by the embodiment of the application, when the data corresponding to the identity is accessed, a plurality of databases do not need to be accessed, and only the block chain corresponding to the identity needs to be accessed and the data is stored through the key pair, so that the data security is ensured, the complexity of the operation can be simplified, and the data access efficiency is improved. According to the scheme, data is stored in a block chain in a full-copy mode, when a block chain link point fails, any node which does not fail can continue to work, and the failed block chain node can be repaired through a normal node. However, the storage overhead of the chain in the form of full copy is too large, and in a system with n nodes, each piece of data is stored by n pieces, so that storing cold data (such as log records and the like) which is rarely accessed in this way causes too much storage space-loss waste.
Disclosure of Invention
The invention provides a high-reliability low-overhead data storage system and method for block chain log storage, aiming at overcoming the defects that the existing log storage scheme is high in overhead and easy to generate waste of storage resources.
The primary objective of the present invention is to solve the above technical problems, and the technical solution of the present invention is as follows:
the invention provides a high-reliability low-overhead data storage system for block chain log storage in a first aspect, which comprises: the log client is in communication connection with the log alliance link node through an RPC interface, the log client is used for encoding a log file to obtain an encoding block, and the log alliance link node is used for storing metadata of the encoding block and the log file.
Further, the log client includes: the system comprises an erasure code engine, a blockchain light client and a first RPC interface, wherein the erasure code engine is used for performing erasure code encoding on a log file, the blockchain light client is used for verifying existence of transaction, and the first RPC interface is used for interacting with log federation link point data.
Further, the erasure code engine encodes the log file by the following process:
each log file is divided into n-2f data blocks, and 2f check blocks are generated after (n-2f,2f) -RS encoding; the n-2f data blocks and the 2f check blocks are collectively called as coding blocks, then the n-2f coding blocks are uploaded to nodes of the alliance chain, n represents the total number of the nodes, and f represents the tolerable number of malicious nodes.
Further, the block head of the light and medium block chain client keeps synchronous with the whole node, the block head stores the Mercker tree root, when the light client inquires a certain transaction from the whole node, the whole node returns the transaction and the Mercker certification of the transaction, and the existence of the transaction can be verified through the Mercker certification.
Further, the log federation nexus comprises: the log storage layer is used for storing the coding blocks, the block chain layer is used for storing metadata of the log, and the second RPC interface is used for interacting with the log client data; wherein the block chain layer comprises: the system comprises a PBFT consensus engine, a transaction execution module and a P2P network, wherein the PBFT consensus engine is used for executing the existing PBFT consensus algorithm, the transaction execution module is used for collecting transactions and packaging transaction modules, and the P2P network is a peer-to-peer network formed by distributed independent block chain nodes in a block chain.
The second aspect of the present invention provides a high-reliability low-overhead data storage method for blockchain log storage, which is applied to the high-reliability low-overhead data storage system for blockchain log storage, and is characterized by comprising the following steps:
dividing a log file to be stored into two storage stages, wherein the two storage stages comprise: a preparation phase and a submission phase;
and storing the log file into the block chain log storage system in stages.
Further, in the preparation stage, the block chain protocol does not work, the block chain server serves as a storage server, the client sends the coding block to the server, and the signature returned by the server is stored;
in the commit phase, the client sends a commit request to the blockchain, and attaches a signature returned by the preparation phase server to the request, and the blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.
Further, in the above-mentioned case,
in the first stage, the alliance chain is logically a down-chain storage node and runs a protocol of a log storage layer; after the log client encodes the log file and uploads the log file to the alliance chain node, the alliance chain node checks the encoding block and stores the encoding block in a KV database on the node, after successful storage, the node signs the encoding block by using a private key of the node, and the signature is returned and stored in the block chain to prove that the node successfully stores the log encoding block.
Further, after the log client collects enough signatures after more than n-f signatures, it indicates that at least n-2f non-malicious nodes have normally stored the log coding blocks, at this time, the log client can initiate a transaction, and uploads the signatures and other log file metadata to the log alliance chain in a transaction form, the block chain node will check the submitted transaction, and after the check is completed, the signature and other log file metadata are packaged and added in the block chain, and meanwhile, a state database is used for indexing the log file;
when a user inquires a specific log through a client, metadata is firstly obtained from any block chain, and then n-2f correct coding blocks are obtained through a log storage layer and then the log is decoded and recovered at the client.
Further, the client log file storage process specifically comprises the following steps:
a1: using a single log file as an input parameter, encoding data of the log file by using an erasure coding function to obtain coding blocks with the number equal to that of clusters, and calculating a checksum through hash operation;
a2: the alliance chain node Ni is sequenced from 1 to n according to the public key, the client sends a corresponding coding block Ci, the verification of the log file and the verification of the coding block Ci to the alliance chain node Ni, and n represents the number of clusters;
a3: the alliance link node checks whether the coding block is consistent with the check; ensuring that the log file corresponding to the coding block is not submitted;
a4: defining a key value pair, taking the check of a log file in a coding block as a key, storing the coding block on local object data of a node, and taking data associated with the log file as a value, wherein the value comprises data of the coded block file, check information and metadata covering client information, a log timestamp and application program path information, and the condition that the log file is in a preparation stage and is not submitted is represented when the value of the metadata is empty;
a5: the alliance link node returns the check information of the log file and the signature of the encoded file check information to indicate that the encoding block is received and stored, and the client submits the signature as a certificate normally received by the node at the next stage;
a6: detecting whether a client collects a preparation completion message returned by more than n-2f alliance link points, and if so, awakening a main thread;
the a7 client submits the log file metadata and the signature returned by the federation chain node as a transaction.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
the invention completes the storage of the log file by the submission of the log file into a preparation stage and a submission stage, can effectively reduce the storage overhead and expand the storage capacity of the system by multiplexing the physical node where the block chain system is located as the down-chain storage node under the condition of not increasing additional machines, and simultaneously avoids the deletion or the tampering of the data by the Byzantine node by combining the redundancy capability provided by the erasure coding.
Drawings
Fig. 1 is a block diagram of a high-reliability low-overhead data storage system for blockchain log storage according to the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
Example 1
As shown in fig. 1, the present invention provides a high-reliability low-overhead data storage system for blockchain log storage, including: the log client is in communication connection with the log alliance link node through an RPC interface, the log client is used for encoding a log file to obtain an encoding block, and the log alliance link node is used for storing metadata of the encoding block and the log file.
In the embodiment of the invention, the log file is encoded at the client to obtain the encoding block, and the encoding block is uploaded to the log alliance node for storage; when enough nodes return messages which are successfully stored, the log metadata is used as transactions and sent to the block chain; and when the incoming data is read, the client sends a request to the alliance chain node to read the coding block and recovers the log file at the client.
Further, the log client comprises: the system comprises an erasure code engine, a blockchain light client and a first RPC interface, wherein the erasure code engine is used for performing erasure code encoding on a log file, the blockchain light client is used for verifying existence of transaction, and the first RPC interface is used for interacting with log federation link point data.
Further, the erasure code engine encodes the log file by the following process:
each log file is divided into n-2f data blocks, and 2f check blocks are generated after (n-2f,2f) -RS encoding; the n-2f data blocks and the 2f check blocks are collectively called as coding blocks, then the n-2f coding blocks are uploaded to nodes of the alliance chain, n represents the total number of the nodes, and f represents the tolerable number of malicious nodes.
It should be noted that the client saves the log file to the offline storage before submitting the log metadata to the blockchain in the form of transaction. In the embodiment of the invention, in order to increase the reliability of the storage under the chain, the peer nodes in the alliance chain are used as the nodes for the storage under the chain, and the log file is written in the form of erasure code coding blocks. The log chain of the novel domain name resolution system uses the PBFT as a consensus protocol, a distributed system using the PBFT can tolerate no more than 1/3 malicious nodes, and when the total number of nodes is n, if f malicious nodes are to be tolerated, n is required to be greater than or equal to 3f + 1. When a piece of data successfully submitted through the PBFT protocol is read, the correct result can be read only on n-2f nodes. For example, if f malicious nodes pretend to be normal nodes when the message m is submitted through the protocol, if f non-malicious nodes in the system fail and go down, the system can still complete the submission and agreement of the message m, and at this time, the message m is written into n-f nodes, wherein n-2f are normal nodes and the other f are malicious nodes. When the message m is read, the malicious node can return an error message, so that only correct information can be read from n-2f nodes. In the invention, the maximum number f of Byzantine nodes is determined and configured before operation, which determines the erasure code coding mode used later, and the client uses (n-2f,2f) -RS to code each log file. Specifically, each log file is divided into n-2f data blocks, and 2f check blocks are generated after (n-2f,2f) -RS encoding; the n-2f data blocks and the 2f check blocks are collectively called as coding blocks, and then the n-2f coding blocks are uploaded to the nodes of the alliance chain (stored under the chain); during reading, the log file can be recovered only by acquiring n-2f correct coding blocks.
Further, the block head of the light and medium block chain client keeps synchronous with the whole node, the block head stores the Mercker tree root, when the light client inquires a certain transaction from the whole node, the whole node returns the transaction and the Mercker proof of the transaction, and the existence of the transaction can be verified through the Mercker proof.
In the embodiment of the invention, the log is uploaded and downloaded in the client of the log block chain. The client in the invention interacts with the block chain link points through a light client protocol. Different from the full node, the light client does not participate in ore excavation and does not store the block bodies, and the requirements on the computing capacity and the storage capacity are lower than those of the full node. When the light client inquires a certain transaction from the whole node, the whole node returns the transaction and a Mercker certificate of the transaction, and the existence of the transaction can be verified through the Mercker certificate. The light client protocol enables a user to participate in the block chain at a small cost, and the load of the block chain network is reduced because the user only needs to acquire the block head during synchronization.
In the present invention, the log federation chain node includes: the log storage layer is used for storing the coding blocks, the block chain layer is used for storing metadata of the log, and the second RPC interface is used for interacting with the log client side data.
In an embodiment of the present invention, the block chain layer includes: the system comprises a PBFT consensus engine, a transaction execution module and a P2P network, wherein the PBFT consensus engine is used for executing the existing PBFT consensus algorithm, the transaction execution module is used for collecting transactions and packaging transaction modules, and the P2P network is a peer-to-peer network formed by distributed independent block chain nodes in a block chain.
Example 2
The invention also provides a high-reliability low-overhead data storage method for the block chain log storage, which is applied to the high-reliability low-overhead data storage system for the block chain log storage and comprises the following steps:
dividing a log file to be stored into two storage stages, wherein the two storage stages comprise: a preparation phase and a submission phase;
and storing the log file into the block chain log storage system in stages.
Further, in the preparation stage, the blockchain protocol does not work, the blockchain server serves as a storage server, the client sends the coding block to the server, and the signature returned by the server is stored;
in the commit phase, the client sends a commit request to the blockchain, and attaches a signature returned by the preparation phase server to the request, and the blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.
Furthermore, in the two stages of storage of the log file, the nodes of the alliance chain play two different roles in the two stages, and in the first stage, the nodes of the alliance chain are logically linked storage nodes and run a protocol of a log storage layer; after the log client encodes the log file and uploads the log file to the alliance chain node, the alliance chain node checks the encoding block and stores the encoding block in a KV database on the node, after successful storage, the node signs the encoding block by using a private key of the node, and the signature is returned and stored in the block chain to prove that the node successfully stores the log encoding block.
Further, after the log client collects enough signatures after more than n-f signatures, it indicates that at least n-2f non-malicious nodes have normally stored the log coding blocks, at this time, the log client can initiate a transaction, and uploads the signatures and other log file metadata to the log alliance chain in a transaction form, the block chain node will check the submitted transaction, and after the check is completed, the signature and other log file metadata are packaged and added in the block chain, and meanwhile, a state database is used for indexing the log file;
when a user inquires a specific log through a client, metadata is firstly obtained from any block chain, and then n-2f correct coding blocks are obtained through a log storage layer and then the log is decoded and recovered at the client.
Example 3
In the invention, the specific steps of storing the log file of the client are as follows:
a1: using a single log file as an input parameter, encoding data of the log file by using an erasure coding function to obtain coding blocks with the number equal to that of clusters, and calculating a checksum through hash operation;
a2: the alliance chain node Ni is sequenced from 1 to n according to the public key, the client sends a corresponding coding block Ci, the verification of the log file and the verification of the coding block Ci to the alliance chain node Ni, and n represents the number of clusters;
a3: the alliance link node checks whether the coding block is consistent with the check; ensuring that the log file corresponding to the coding block is not submitted;
a4: defining a key value pair, taking the check of a log file in a coding block as a key, storing the coding block on local object data of a node, and taking data associated with the log file as a value, wherein the value comprises data (marked as d) of a coded block (decoded-chunk) file, check information (marked as dc), and metadata (marked as meta) covering client information, a log timestamp and application program path information, and the metadata value is empty and indicates that the log file is in a preparation stage and is not submitted;
a5: the federation link node returns a signature for log file verification and file verification after encoding to indicate that the encoding block is received and stored, and the client submits the signature as a certificate normally received by the node at the next stage;
a6: detecting whether a client collects a preparation completion message returned by more than n-2f alliance link points, and if so, awakening a main thread;
a7: the client submits the log file metadata and the signature returned by the federation chain node as a transaction.
It should be noted that, in the commit phase, the commit message includes: the log file metadata comprises client information, log time range, application program path and other information, and can be used for retrieving logs, and the log file checksum and the signature returned by the federation link node can be used for verifying the authenticity of the log file. More specifically, a transaction submitted by a client calls a check _ tx function before entering a memory pool, the check _ tx function checks the number of correct signatures, the SUCCESS is returned after the coding blocks are temporarily stored by more than 2f nodes, and the transaction enters the memory pool and is submitted through a PBFT protocol. After completion of the commit and after packing the blocks, each blockchain node will call the deliverer _ tx function. The deliverer _ tx function is used for updating a state machine in a block chain, the update state machine writes metadata of the log file, the metadata is written in order to facilitate a client to inquire the corresponding log file through the metadata, and the successful formal submission of the log file is also shown.
It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (10)
1. A high reliability low overhead data storage system oriented to blockchain log storage, comprising: the log client is in communication connection with the log alliance link node through an RPC interface, the log client is used for encoding a log file to obtain an encoding block, and the log alliance link node is used for storing metadata of the encoding block and the log file.
2. The method as claimed in claim 1, wherein the log client comprises: the system comprises an erasure code engine, a blockchain light client and a first RPC interface, wherein the erasure code engine is used for performing erasure code encoding on a log file, the blockchain light client is used for verifying existence of transaction, and the first RPC interface is used for interacting with log federation link point data.
3. The system according to claim 2, wherein the erasure coding engine encodes the log file by:
each log file is divided into n-2f data blocks, and 2f check blocks are generated after (n-2f,2f) -RS encoding; the n-2f data blocks and the 2f check blocks are collectively called as coding blocks, then the n-2f coding blocks are uploaded to nodes of the alliance chain, n represents the total number of the nodes, and f represents the tolerable number of malicious nodes.
4. The system of claim 3, wherein a block head of the blockchain light client keeps synchronous with a whole node, and a Merck tree root is stored in the block head, when the light client queries the whole node for a transaction, the whole node returns the transaction and a Merck proof of the transaction, and the existence of the transaction can be verified through the Merck proof.
5. The method for high-reliability low-overhead data storage oriented to blockchain log storage of claim 1, wherein the log federation chain node comprises: the log storage layer is used for storing the coding blocks, the block chain layer is used for storing metadata of the log, and the second RPC interface is used for interacting with the log client data; wherein the block chain layer comprises: the system comprises a PBFT consensus engine, a transaction execution module and a P2P network, wherein the PBFT consensus engine is used for executing the existing PBFT consensus algorithm, the transaction execution module is used for collecting transactions and packaging transaction modules, and the P2P network is a peer-to-peer network formed by distributed independent block chain nodes in a block chain.
6. A method for storing high-reliability low-overhead data for blockchain log storage, which is applied to the high-reliability low-overhead data storage system for blockchain log storage according to any one of claims 1 to 5, and is characterized by comprising the following steps:
dividing a log file to be stored into two storage stages, wherein the two storage stages comprise: a preparation phase and a submission phase;
and storing the log file into the block chain log storage system in stages.
7. The method for storing high-reliability low-overhead data oriented to blockchain log storage of claim 6, wherein in a preparation phase, the blockchain protocol does not work, the blockchain server acts as a storage server, the client sends the coding blocks to the server, and saves a signature returned by the server;
in the commit phase, the client sends a commit request to the blockchain, and attaches a signature returned by the preparation phase server to the request, and the blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.
8. The method of claim 7, wherein the high reliability and low overhead data storage method for blockchain log storage,
in the first stage, the alliance chain is logically a down-chain storage node and runs a protocol of a log storage layer; after the log client encodes the log file and uploads the log file to the alliance chain node, the alliance chain node checks the encoding block and stores the encoding block in a KV database on the node, after successful storage, the node signs the encoding block by using a private key of the node, and the signature is returned and stored in the block chain to prove that the node successfully stores the log encoding block.
9. The method as claimed in claim 8, wherein when the log client collects enough signatures after more than n-f signatures, it indicates that at least n-2f non-malicious nodes have normally saved log coding blocks, and at this time, the log client can initiate a transaction, upload the signatures and other log file metadata to the log federation chain in the form of a transaction, and the block chain node will check the submitted transaction, and package and add the transaction in the block chain after the check is completed, and there is a status database to index the log file;
when a user inquires a specific log through a client, metadata is firstly obtained from any block chain, and then n-2f correct coding blocks are obtained through a log storage layer and then the log is decoded and recovered at the client.
10. The method for storing high-reliability low-overhead data oriented to blockchain log storage according to claim 9, wherein the specific steps of the client log file storage process are as follows:
a1: using a single log file as an input parameter, encoding data of the log file by using an erasure coding function to obtain coding blocks with the number equal to that of clusters, and calculating a checksum through hash operation;
a2: the alliance chain node Ni is sequenced from 1 to n according to the public key, the client sends a corresponding coding block Ci, the verification of the log file and the verification of the coding block Ci to the alliance chain node Ni, and n represents the number of clusters;
a3: the alliance link node checks whether the coding block is consistent with the check; ensuring that the log file corresponding to the coding block is not submitted;
a4: defining a key value pair, taking the check of a log file in a coding block as a key, storing the coding block on local object data of a node, and taking data associated with the log file as a value, wherein the value comprises data of the coded block file, check information and metadata covering client information, a log timestamp and application program path information, and the condition that the log file is in a preparation stage and is not submitted is represented when the value of the metadata is empty;
a5: the alliance link node returns the check information of the log file and the signature of the encoded file check information to indicate that the encoding block is received and stored, and the client submits the signature as a certificate normally received by the node at the next stage;
a6: detecting whether a client collects a preparation completion message returned by more than n-2f alliance link points, and if so, awakening a main thread;
the a7 client submits the log file metadata and the signature returned by the federation chain node as a transaction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111333973.3A CN113986143A (en) | 2021-11-11 | 2021-11-11 | Block chain log storage-oriented high-reliability low-overhead data storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111333973.3A CN113986143A (en) | 2021-11-11 | 2021-11-11 | Block chain log storage-oriented high-reliability low-overhead data storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113986143A true CN113986143A (en) | 2022-01-28 |
Family
ID=79748016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111333973.3A Pending CN113986143A (en) | 2021-11-11 | 2021-11-11 | Block chain log storage-oriented high-reliability low-overhead data storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113986143A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114594911A (en) * | 2022-03-13 | 2022-06-07 | 西安电子科技大学 | Block chain data storage system and method based on under-chain erasure code distributed storage |
| CN115499453A (en) * | 2022-06-28 | 2022-12-20 | 重庆邮电大学 | Sharding storage method facing alliance chain |
| CN116028990A (en) * | 2023-03-30 | 2023-04-28 | 中国科学技术大学 | Anti-tampering privacy protection log auditing method based on blockchain |
-
2021
- 2021-11-11 CN CN202111333973.3A patent/CN113986143A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114594911A (en) * | 2022-03-13 | 2022-06-07 | 西安电子科技大学 | Block chain data storage system and method based on under-chain erasure code distributed storage |
| CN114594911B (en) * | 2022-03-13 | 2024-03-29 | 西安电子科技大学 | Block chain data storage system and method based on under-chain erasure code distributed storage |
| CN115499453A (en) * | 2022-06-28 | 2022-12-20 | 重庆邮电大学 | Sharding storage method facing alliance chain |
| CN115499453B (en) * | 2022-06-28 | 2024-03-12 | 重庆邮电大学 | Fragment storage method oriented to alliance chain |
| CN116028990A (en) * | 2023-03-30 | 2023-04-28 | 中国科学技术大学 | Anti-tampering privacy protection log auditing method based on blockchain |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI737392B (en) | Computer-implemented method for processing blockchain data by a blockchain node of a blockchain network in a trusted execution environment (tee), system communicating shared blockchain data and apparatus for communicating shared blockchain data | |
| CN107193490B (en) | Distributed data storage system and method based on block chain | |
| CN113986143A (en) | Block chain log storage-oriented high-reliability low-overhead data storage method | |
| CN111177080B (en) | Knowledge graph storage and verification method based on block chain and IPFS | |
| TWI729880B (en) | Shared blockchain data storage based on error correction coding in trusted execution environments | |
| TWI720918B (en) | Consenus of shared blockchain data storage based on error correction code | |
| JP7050955B2 (en) | Prioritize storage of shared blockchain data | |
| TWI759791B (en) | Method, system and apparatus of shared blockchain data storage based on error correction code | |
| JP7159348B2 (en) | Dynamic Blockchain Data Storage Based on Error Correcting Codes | |
| CN110555783A (en) | block chain-based power marketing data protection method and system | |
| CN116827957B (en) | Information processing method, device, equipment and medium based on multi-block chain | |
| CN113779146A (en) | Distributed electronic certificate verifiable storage system based on block chain | |
| CN113961149B (en) | Polymorphic data storage system and method for electric power information system | |
| CN116760632B (en) | Data processing method, device, equipment and readable storage medium | |
| CN117172913B (en) | Intelligent contract-based contract change procedure execution method and system | |
| Mahajan et al. | Astro: Autonomous and trustworthy data sharing | |
| Wang et al. | A Trustworthy Data Verification Technique for Cross-Chain Data Sharing Based on Merkle Trees | |
| CN113849583A (en) | Block chain implementation method based on relational database | |
| CN117155953A (en) | Data processing method, device, computer equipment and readable storage medium | |
| CN112929402A (en) | Mobile ad hoc network device for realizing trusted data storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |