CN111008218A - Method and system for structured storage of data on block chain - Google Patents
Method and system for structured storage of data on block chain Download PDFInfo
- Publication number
- CN111008218A CN111008218A CN201911225763.5A CN201911225763A CN111008218A CN 111008218 A CN111008218 A CN 111008218A CN 201911225763 A CN201911225763 A CN 201911225763A CN 111008218 A CN111008218 A CN 111008218A
- Authority
- CN
- China
- Prior art keywords
- data
- block
- height
- structured
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/382—Payment protocols; Details thereof insuring higher security of transaction
- G06Q20/3827—Use of message hashing
Abstract
The present application relates to the field of blockchain technologies, and in particular, to a method and a system for structured storage of data on a blockchain. The method comprises the following steps: the data detection module acquires a block from the block chain according to the maximum block height, the current processing height and the block height flag bit, sets a first flag bit of the corresponding height of the block, and sends the block to the data filtering module; the data filtering module analyzes the required transaction record into structured data and sends the structured data to the data storage module; the data storage module stores the structured data; the data detection module sets a second zone bit with the corresponding height of the block and simultaneously performs an operation of adding one to the current processing height; and repeatedly executing the steps to realize the structured storage of all block data on the block chain. The method can completely acquire the data on the block chain in real time, simultaneously support the structural conversion of data in various different formats, and provide a flexible data storage expansion mode and an efficient data query mode.
Description
Technical Field
The present application relates to the field of blockchain technologies, and in particular, to a method and a system for structured storage of data on a blockchain.
Background
The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. The Blockchain (Blockchain) is an important concept of bitcoin, which is essentially a decentralized database, and is used as the underlying technology of bitcoin, and is a series of data blocks which are associated by using a cryptographic method, wherein each data block contains information of a batch of bitcoin network transactions, and the information is used for verifying the validity of the information and generating the next block.
As the quantity of data carried by the blockchain network increases with time, the variety of data is also various. In this case, the storage and retrieval of the data on the chain mainly have the following problems:
1. data retrieval on the chain is difficult. The storage of data by bitcoin networks is at the core of transactions and not at the core of the services of interest, which, in addition to the increase in data volume, leads to difficulties in data retrieval on the chain.
2. It is difficult to store the data on the chain in a structured manner. If a large block stores 2G data, and one block is found every ten minutes on average, 288G data is available in one day, and as more and more enterprises chain their respective data, the kinds of data on the chain are more and more, and the structuring manner of each data is different, which causes inconvenience in storing structured data.
3. How to handle block chain data rollback situations. In the block generation process of the block chain, different nodes may simultaneously generate a next block in a short time and broadcast the next block to the block chain, and other nodes continue to work on the basis of the first received block, so that the block chain has different next blocks at the same height, and the block chain is temporarily branched. With the continuous generation of subsequent blocks, the block chain takes the longest branched chain as a main chain according to the longest chain principle, all nodes continue to work on the basis, the abandoned branched chain becomes a side chain, and the blocks on the side chain are called as solitary blocks. When a lone block occurs, data on a chain changes, which causes the data which is stored in a structured mode to be correspondingly rolled back and updated, but a good processing mechanism does not exist at present.
4. How to ensure the consistency of data in the data structured storage process. Data is stored in blocks on a chain, and in the process of structured storage of data on the chain, a process may be interrupted for various reasons (such as power failure), and a set of mechanism is required to ensure that acquired data is not repeated or missing.
Disclosure of Invention
One of the objectives of the present invention is to overcome the above disadvantages and provide a method for structured storage of data on a blockchain, which can completely acquire data on the blockchain in real time, support structured conversion of data in multiple different formats, and provide a flexible data storage expansion manner and an efficient data query manner.
In order to solve the above technical problem, the present invention provides a method for storing data in a block chain in a structured manner, which includes the following steps:
step 1, a data detection module acquires the current maximum block height of a block chain;
step 3, the data detection module judges whether the current processing height is less than or equal to the maximum block height, then the data detection module jumps to step 5 to continue execution, and if not, the data detection module executes step 4;
step 6, the data detection module sets a first zone bit of the corresponding height of the block and sends the block to the data filtering module;
step 7, the data filtering module checks the transaction records in the block, analyzes the required transaction records into structured data and sends the structured data to the data storage module;
and repeatedly executing the steps to realize the structured storage of all block data on the block chain.
According to the method, the data detection module controls the integrity of processing by adopting a method of setting double zone bits for the height of one block, the data filtering module is responsible for carrying out targeted structurization on the needed chain data, the data storage module carries out unified storage management control, the structured storage of the block chain data is realized through the division and cooperation of the three modules, and the deployment is flexible and convenient.
Further, the "acquiring a block from a block chain by the data detection module according to the current processing height and the block height flag bit" specifically includes:
and step 55, the data detection module acquires the block corresponding to the current processing height from the block chain.
By comparing the two flag bits of the block height, it can be known that the processing condition of a certain block is completed or not completed, and if the last processing is not completed due to some reasons, the transaction data corresponding to the block height is cleared and reprocessed to ensure the correctness and integrity of the data.
Furthermore, the number of the data filtering modules is multiple, each data filtering module receives the block sent by the data detection module, analyzes the required transaction record into structured data, and sends the structured data to the data storage module.
A data filtering module can be set for one data type, so that no matter how many types of structured data are, the support can be achieved through the corresponding data filtering module, the expansibility is strong, and the application scene is more flexible.
Further, the data storage module includes a plurality of database instances, and the data storage module stores the structured data specifically includes: the data storage module randomly writes the structured data into a database instance.
The data storage module solves the problem of large data volume by setting a plurality of database instances and adopting a random insertion mode. Random insertion supports unlimited capacity expansion, and once the current database instance cannot support the capacity of data, the database instance can be increased at will.
Further, the method for storing the data structure on the blockchain further comprises the following steps:
the data detection module acquires an unconfirmed transaction record from a memory pool of the block chain and sends the unconfirmed transaction record to the data filtering module;
the data filtering module analyzes the required unconfirmed transaction record into structured data and sends the structured data to the data storage module.
By acquiring the transaction data which is not written into the block chain in the memory pool, the zero confirmation transaction is supported, and the transaction synchronization and the query are more real-time.
Further, the method for storing the data structure on the blockchain further comprises the following steps:
the data detection module checks whether the hash value of the processed block is consistent with the hash value of the corresponding height block on the current block chain at regular time, and if not, the second flag bit with the corresponding height in the block height flag bits is cleared.
By controlling the second zone bit of the block height, the transaction data can be automatically returned and updated without manual intervention when the isolated block appears.
Further, the method for storing the data structure on the blockchain further comprises the following steps:
and the data storage module performs concurrent query on the structured data in all database instances and summarizes query results.
The data storage module realizes quick retrieval under the condition of large data volume by setting a plurality of database instances and carrying out concurrent unified query, thereby improving the query performance.
Correspondingly, the application also provides a data structured storage system on the block chain, which comprises a data detection module, a data filtering module and a data storage module, wherein,
the data detection module is used for acquiring the current maximum block height of the block chain; setting an initial value of a current processing height, and acquiring a block from a block chain according to the current processing height and a block height flag bit; the block height flag bit comprises a first flag bit and a second flag bit of the corresponding height of each block on the block chain; the first flag bit is used for setting the corresponding height of the block and sending the block to a data filtering module; the second flag bit is used for setting the corresponding height of the block after the block processing is finished, and simultaneously adding one to the current processing height;
the data filtering module is used for checking the transaction records in the block, analyzing the required transaction records into structured data and sending the structured data to the data storage module;
and the data storage module is used for storing the structured data sent by the data filtering module.
Further, the data detection module is further configured to obtain an unconfirmed transaction record from a memory pool of the block chain and send the unconfirmed transaction record to the data filtering module; the data filtering module is also used for analyzing the needed unconfirmed transaction records into structured data and sending the structured data to the data storage module.
Further, the data detection module is further configured to check whether the hash value of the processed block is consistent with the hash value of the corresponding height block in the current block chain at regular time, and if not, clear the second flag bit of the corresponding height in the block height flag bits.
Different from the prior art, the technical scheme of the invention has the following beneficial effects:
1. according to the method, the data detection module controls the integrity of processing by adopting a method of setting double zone bits for the height of one block, the data filtering module is responsible for carrying out targeted structurization on the needed chain data, the data storage module carries out unified storage management control, the structured storage of the block chain data is realized through the division and cooperation of the three modules, and the deployment is flexible and convenient.
2. By comparing the two flag bits of the block height, it can be known that the processing condition of a certain block is completed or not completed, and if the last processing is not completed due to some reasons, the transaction data corresponding to the block height is cleared and reprocessed to ensure the correctness and integrity of the data. Meanwhile, by controlling the second zone bit of the block height, the transaction data can be automatically returned and updated without manual intervention when the isolated block appears.
3. A data filtering module can be set for one data type, so that no matter how many types of structured data are, the support can be achieved through the corresponding data filtering module, the expansibility is strong, and the application scene is more flexible.
4. The data storage module solves the problem of large data volume by setting a plurality of database instances and adopting a random insertion mode. Random insertion supports unlimited capacity expansion, and once the current database instance cannot support the capacity of data, the database instance can be increased at will. Meanwhile, rapid retrieval under the condition of large data volume is realized by concurrent unified query, and the query performance is improved.
5. By acquiring the transaction data which is not written into the block chain in the memory pool, the zero confirmation transaction is supported, and the transaction synchronization and the query are more real-time.
Drawings
Fig. 1 is a flowchart illustrating steps of a method for storing data in a block chain in a structured manner according to the present invention.
FIG. 2 is a flowchart of the steps taken by the data detection module to obtain a block from the chain of blocks according to the current process height and the block height flag.
Fig. 3 is an architecture diagram of a data structured storage system on a blockchain according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data on the whole block chain is stored on all the nodes in a block form, and the block chain data structured storage method acquires the data of all the blocks on the block chain, performs structured conversion after filtering, and then stores and persists the data.
Referring to fig. 1, a flow chart of steps of a method for storing data in a structured manner on a blockchain according to the present invention includes the following steps:
step 1, a data detection module acquires the current maximum block height of a block chain;
in the invention, the data detection module is responsible for acquiring blocks needing to be processed from all nodes of the block chain and forwarding the blocks to the data filtering module. Here, the block height refers to the sequential number of each block in the block chain, and increases with the number of blocks. Since the blocks in the block chain are continuously increased, the data detection module acquires the current maximum block height of the block chain each time to determine whether new block data needs to be processed.
Step 3, the data detection module judges that if the current processing height is less than or equal to the maximum block height, it indicates that newly-added blocks on the block chain are not processed, and then the data detection module jumps to step 5 to continue execution, otherwise, if the current processing height is greater than the maximum block height, it indicates that all blocks on the block chain have been processed, and then step 4 is executed;
4, the data detection module skips to the step 1 after waiting for the set time and continues to execute the next round of operation;
Referring to fig. 2, it is a flowchart of the steps of the data detection module of the present invention obtaining a block from the block chain according to the current processing height and the block height flag bit, and the method includes the following steps:
and step 55, the data detection module acquires the block corresponding to the current processing height from the block chain.
The method comprises the steps that a data detection module controls the integrity of processing by adopting a method of setting double zone bits for the height of a block, whether the processing of the block with a certain height is finished or not can be known by comparing the two zone bits of the height of the block, and if the processing of the block with a certain height is not finished for the last time due to certain reasons, transaction data corresponding to the height of the block are removed and reprocessed to ensure the correctness and the integrity of the data.
Step 6, the data detection module sets a first flag bit of the corresponding height of the block, indicates that the block with the height starts to be processed, and sends the block to the data filtering module;
step 7, the data filtering module checks the transaction records in the block, analyzes the required transaction records into structured data and sends the structured data to the data storage module;
in the application, the data filtering module is a key for customizing the transaction data on the block chain, acquires the data of the whole block from the data detecting module, and filters and structures the transaction data. The data filtering module is essentially a transaction observer, which checks each transaction contained in the block one by one, judges whether one transaction is a transaction record concerned by itself, if the transaction is found to be a transaction needing attention, analyzes and converts the original transaction record data into structured data, wherein the structured data is also called row data, is data logically expressed and realized by a two-dimensional table structure, strictly follows the data format and length specification, and is mainly stored and managed by a relational database.
In a preferred embodiment, the number of the data filtering modules may be multiple, and by adding the data filtering module number identifier to the block height flag bit, the data detection module may identify to which height block each data filtering module has processed respectively, and send the next block to be processed. And each data filtering module analyzes the received transaction records required in the block into structured data and sends the structured data to the data storage module, and the data filtering modules do not interfere with each other and run in parallel. By arranging one data filtering module for each data type to be opposite to the data filtering module, no matter how many types of structured data, the support can be achieved through the corresponding data filtering module, the expansibility is strong, and the application scene is more flexible.
in the application, the data storage module is responsible for unified storage management of all structured data, and persists the structured data. In particular embodiments, a database such as mysql or mongodb may be used to store the transaction data. With the arrival of the big block era, the data of the block chain network becomes larger and larger, and the data storage module serving as a persistence layer needs to be capable of supporting capacity increase. Therefore, in a preferred embodiment, the data storage module includes a plurality of database instances, and when the data storage module stores the structured data, one database instance is randomly selected and the structured data is written into the database instance. In a specific strategy, alternate insertion can be selected, a first piece of data is inserted into a first library, a second library, and the database is restarted after all the libraries are rotated once; or taking hash for the transaction, and complementing the number of the databases by the hash value, wherein the number of the databases is several; and when the storage is insufficient and the new database instances are expanded, all data are only stored in the newly added database instances within a period of time, and other balanced strategies are adopted until the data volume in each database instance is not much. In addition, if the user needs to query the concerned transaction, the data storage module can perform concurrent query of the structured data in all database instances, and summarize the query results and return the summarized query results to the user. By setting a plurality of database instances, unlimited capacity expansion can be supported by adopting a random insertion mode, and once the current database instance cannot support the capacity of data, the database instances can be increased at will. Meanwhile, rapid retrieval under the condition of large data volume is realized by concurrent unified query, and the query performance is improved.
by continuously and repeatedly executing the steps, the data detection module, the data filtering module and the data storage module are in work sharing cooperation, and all block data are continuously and structurally stored along with the increase of block chain blocks.
In a preferred embodiment, the method for storing data structured on a blockchain of the present application further includes the following steps: the data detection module acquires an unconfirmed transaction record from a memory pool of the block chain and sends the unconfirmed transaction record to the data filtering module; the data filtering module analyzes the required unconfirmed transaction record into structured data and sends the structured data to the data storage module. The block chain stores the unconfirmed transaction records in the memory pool of all nodes before the new block is generated, so that the data detection module can monitor the transaction records in the memory pool in real time, directly sends the transaction records to the data filtering module once the new transaction records are generated, judges whether the transaction records are concerned transactions by the data filtering module, and carries out structuralization and sends the transaction records to the data storage module for persistence if the transaction records are concerned transactions by the data filtering module. By acquiring the transaction data which is not written into the block chain in the memory pool, zero confirmation transaction can be supported, and transaction synchronization and subsequent query are more real-time.
In another preferred embodiment, the method for storing data structured on a blockchain of the present application further includes the following steps: the data detection module checks whether the hash value of the processed block is consistent with the hash value of the corresponding height block on the current block chain at regular time, and if not, the second flag bit with the corresponding height in the block height flag bits is cleared.
The hash value information of the current block is stored in the block head of each block in the block chain, the hash value is calculated and generated by all transaction records of the block, the hash value of each generated block is kept unchanged, if the hash value of the processed block is inconsistent with the hash value of the block with the corresponding height on the block chain, the situation that the block chain is forked and the block is changed according to the longest chain principle is explained, and the original block is changed into an isolated block, so that the block data with the height needs to be reprocessed. In the application, only the second zone bit with the corresponding height needs to be cleared, and the data detection module can reprocess the zone bit when judging next time. By controlling the second zone bit of the block height, the transaction data can be automatically returned and updated without manual intervention when the isolated block appears.
The method for storing data structured on blockchain according to the present application is further described below with a specific embodiment. Firstly, a plurality of instances of the Mongo database are deployed in the network as blockchain data storage areas, and meanwhile, one or more full-node instances of blockchains are deployed in the network. In this embodiment, a transaction record on the blockchain is shown in table 1, and there are one input and three outputs for the transaction. The transaction itself is a data structure, and the data stored on the bitcoin network is usually in tx and block units, i.e. it is easier to find the corresponding transaction and the corresponding block through txid and block hash. The design is that the bitcoin network can better operate, but is not very friendly to some applications built on the bitcoin network.
TABLE 1
Meanwhile, the embodiment includes two data filtering modules, which are called address _ filter and meta _ filter for easy distinction, wherein the address _ filter is a data filtering process for one bitcoin wallet application, and it needs to find the relevant transaction stream of the address concerned by the address from the transaction record of the blockchain. When the address _ filter obtains the transaction record, the address participating in the transaction can be obtained by analyzing the locking script and the unlocking script, and the address _ filter is structured into a data structure (as shown in table 2) which is convenient to query according to the address.
| address | txid | index | isinput | pre-txid | pre-index | value |
| Address 1 | txid3 | 0 | true | txid2 | 1 | 300 |
| |
txid3 | 0 | false | - | - | 100 |
| Address 3 | txid3 | 1 | false | - | - | 200 |
TABLE 2
In addition, the metanet is another protocol on the bitcoin network, which uses a binary group of txid and a public key as a node of the metanet, and the node is generated by a parent node and can sign a child node of the node. The metanet protocol requires that the transaction of a metanet node is recorded, firstly, an input is provided, the input unlocking script is provided with a public key, and a private key corresponding to the public key digitally signs the transaction, wherein the public key is the public key of a father node. Second, there is an output of pkscript starting with the operator op _ return, then putting the metanet flag, i.e. "meta", then the public key of the child node to be signed, called pnode, and finally the parent node's payentxid. This results in a new node consisting of the binary set of txid and pnode of the current transaction. Since metanet is a tree structure composed of nodes, it is also inconvenient to search on bitcoin networks. In this embodiment, the tx data is converted into the data structure shown in table three by the data filter module meta _ filter, so that the tree structure is more conveniently maintained.
TABLE 3
Assuming that, in block height flag bits of the address _ filter of the data filtering module, both a first flag bit and a second flag bit having a block height of 990 are already set to 1, and both a first flag bit and a second flag bit having other block heights are not set (that is, 0), the method for structured storage of data on a block chain according to the present application includes the following steps:
step 1, a data detection module acquires the current maximum block height of a block chain, and the acquired value is assumed to be 1000;
step 3, the data detection module judges that the current processing height 0 is smaller than the maximum block height 1000, and then the data detection module jumps to step 5 to continue execution;
since the first flag bit and the second flag bit of the block height equal to 990 are both already set to 1, the above steps are repeated until the current processing height is 991, and the process jumps to step 55 to continue execution.
and step 55, the data detection module acquires the block corresponding to the current processing height (991) from the block chain.
Step 6, the data detection module sets a first zone bit of the corresponding height (991) of the block and sends the block to the data filtering module;
step 7, the data filtering module checks the transaction records in the block, analyzes the required transaction records into structured data and sends the structured data to the data storage module; the method specifically comprises the following steps: when the address _ filter obtains the transaction data, analyzing each vin analysis locking script firstly, obtaining public key information from the vin analysis locking script, deducing a bitcoin address from the public key, checking whether the address is the address of the wallet user, if so, processing the transaction data and sending the transaction data to a data storage module, then analyzing the locking script of each vout, obtaining the address information from the address information, checking whether the address of the wallet user is the address, and if so, processing the transaction data and sending the transaction data to the data storage module. If not, the data concerned by the current data filtering module is not processed.
and repeatedly executing the steps to realize that the data filtering module address _ filter carries out structured storage on the block data with the height from 991 to 1000 on the block chain, and simultaneously updating the first zone bit and the second zone bit with the block height from 991 to 1000 in the zone bit with the block height to be 1.
Meanwhile, in the block height flag bits of the data filter module metanet _ filter, both the first flag bit and the second flag bit having a block height equal to 890 have been set to 1, the first flag bit having a block height equal to 891 has been set to 1, the second flag bit is 0, and both the first flag bit and the second flag bit having other block heights have not been set (i.e. are 0), the method for structured storage of data on a block chain according to the present application includes the following steps:
step 1, a data detection module acquires the current maximum block height of a block chain, wherein the value is 1000;
step 3, the data detection module judges that the current processing height 800 is smaller than the maximum block height 1000, and then the data detection module jumps to step 5 to continue execution;
since both the first flag and the second flag of the tile height equal to 890 have been set to 1, the above step logic is repeated until the current processing height is set to 891, jumping to step 54 to continue execution.
and step 55, the data detection module acquires the block corresponding to the current processing height (891) from the block chain.
Step 6, the data detection module sets the first flag bit of the corresponding height (891) of the block to be 1 and sends the block to the data filtering module;
step 7, the data filtering module checks the transaction records in the block, analyzes the required transaction records into structured data and sends the structured data to the data storage module; the method specifically comprises the following steps: the data filtering module metanet _ filter first checks whether the vout is started by the op _ return operator, then checks whether the script is followed by a metanet flag, namely 'meta', and then continues to check pnode, parentxid, if these all conform to the format, this is a metanet node, and the data filtering module metanet _ filter structures the message and sends the message to the data storage module.
and repeating the steps to realize the structured storage of the block data with the height from 891 to 1000 on the block chain, and simultaneously updating the first flag bit and the second flag bit of the block height from 891 to 1000 in the block height flag bits to be 1.
It should be noted that, in the above embodiment, the data detection module obtains corresponding blocks from the block chain according to the block height flag bit information of the data filtering modules address _ filter and metanet _ filter, and sends the corresponding blocks to the data filtering modules, each data filtering module performs filtering and structuring of data according to its own mode, and sends the data to the data storage module, and the data processing of the two data detection modules are independent and do not interfere with each other.
Fig. 3 is an architecture diagram of a data structure storage system on a blockchain according to the present invention, which includes a data detection module, a data filtering module and a data storage module, wherein,
the data detection module is used for acquiring the current maximum block height of the block chain; setting an initial value of a current processing height, and acquiring a block from a block chain according to the current processing height and a block height flag bit; the block height flag bit comprises a first flag bit and a second flag bit of the corresponding height of each block on the block chain; the first flag bit is used for setting the corresponding height of the block and sending the block to a data filtering module; the second flag bit is used for setting the corresponding height of the block after the block processing is finished, and simultaneously adding one to the current processing height;
the data filtering module is used for checking the transaction records in the block, analyzing the required transaction records into structured data and sending the structured data to the data storage module; in a preferred embodiment, the number of the data filtering modules may be multiple, and by adding the data filtering module number identifier to the block height flag bit, the data detection module may identify to which height block each data filtering module has processed respectively, and send the next block to be processed. And each data filtering module analyzes the received transaction records required in the block into structured data and sends the structured data to the data storage module, and the data filtering modules do not interfere with each other and run in parallel. By arranging one data filtering module for each data type to be opposite to the data filtering module, no matter how many types of structured data, the support can be achieved through the corresponding data filtering module, the expansibility is strong, and the application scene is more flexible.
And the data storage module is used for storing the structured data sent by the data filtering module. In a preferred embodiment, the data storage module includes a plurality of database instances, and when the data storage module stores the structured data, one database instance is randomly selected and the structured data is written into the database instance. In addition, if the user needs to query the concerned transaction, the data storage module can perform concurrent query of the structured data in all database instances, and summarize the query results and return the summarized query results to the user. By setting a plurality of database instances, unlimited capacity expansion can be supported by adopting a random insertion mode, and once the current database instance cannot support the capacity of data, the database instances can be increased at will. Meanwhile, rapid retrieval under the condition of large data volume is realized by concurrent unified query, and the query performance is improved.
In a preferred embodiment, in the data structured storage system on a blockchain described in the present application, the data detection module is further configured to obtain an unconfirmed transaction record from a memory pool of the blockchain and send the unconfirmed transaction record to the data filtering module; the data filtering module is also used for analyzing the needed unconfirmed transaction records into structured data and sending the structured data to the data storage module.
In another preferred embodiment, in the data structured storage system on a block chain described in the present application, the data detection module is further configured to check whether the hash value of the processed block is consistent with the hash value of the block with the corresponding height on the current block chain at regular time, and if not, clear the second flag bit with the corresponding height in the block height flag bits.
The above embodiments are merely illustrative of the technical solutions of the present invention, and the present invention is not limited to the above embodiments, and any modifications or alterations according to the principles of the present invention should be within the protection scope of the present invention.
Claims (10)
1. A method for storing data structuralization on a block chain is characterized by comprising the following steps:
step 1, a data detection module acquires the current maximum block height of a block chain;
step 2, the data detection module sets the initial value of the current processing height;
step 3, the data detection module judges whether the current processing height is less than or equal to the maximum block height, then the data detection module jumps to step 5 to continue execution, and if not, the data detection module executes step 4;
step 4, the data detection module skips to the step 1 to continue executing after waiting for the set time;
step 5, the data detection module acquires a block from the block chain according to the current processing height and the block height flag bit; the block height flag bit comprises a first flag bit and a second flag bit of the corresponding height of each block on the block chain;
step 6, the data detection module sets a first zone bit of the corresponding height of the block and sends the block to the data filtering module;
step 7, the data filtering module checks the transaction records in the block, analyzes the required transaction records into structured data and sends the structured data to the data storage module;
step 8, the data storage module stores the structured data;
step 9, the data detection module sets a second zone bit of the corresponding height of the block, and simultaneously executes an operation of adding one to the current processing height, and jumps to step 3 to continue execution;
and repeatedly executing the steps to realize the structured storage of all block data on the block chain.
2. The method as claimed in claim 1, wherein said data detection module obtains a block from the blockchain according to the current processing height and the block height flag bit, and comprises:
step 51, the data detection module judges whether the first flag bit of the current processing height is set, then step 52 is executed, otherwise, step 55 is skipped to continue execution;
step 52, the data detection module judges whether the second flag bit of the current processing height is set, then step 53 is executed, otherwise step 54 is skipped to continue execution;
step 53, the data detection module performs an operation of adding one to the current processing height, and jumps to step 3 to continue the execution;
step 54, the data detection module calls the data storage module to delete the transaction record corresponding to the current processing height;
and step 55, the data detection module acquires the block corresponding to the current processing height from the block chain.
3. The method for structured storage of data on a blockchain according to claim 1, wherein the number of data filtering modules is plural, and each data filtering module receives the block sent by the data detecting module, parses the required transaction record into structured data, and sends the structured data to the data storage module.
4. The method according to claim 1, wherein the data storage module comprises a plurality of database instances, and the step of storing the structured data by the data storage module comprises: the data storage module randomly writes the structured data into a database instance.
5. The method for structured storage of data on a blockchain according to claim 1, further comprising the steps of:
the data detection module acquires an unconfirmed transaction record from a memory pool of the block chain and sends the unconfirmed transaction record to the data filtering module;
the data filtering module analyzes the required unconfirmed transaction record into structured data and sends the structured data to the data storage module.
6. The method for structured storage of data on a blockchain according to claim 1, further comprising the steps of:
the data detection module checks whether the hash value of the processed block is consistent with the hash value of the corresponding height block on the current block chain at regular time, and if not, the second flag bit with the corresponding height in the block height flag bits is cleared.
7. The method for structured storage of data on a blockchain according to claim 3, further comprising the steps of:
and the data storage module performs concurrent query on the structured data in all database instances and summarizes query results.
8. A data structured storage system on a block chain is characterized by comprising a data detection module, a data filtering module and a data storage module, wherein,
the data detection module is used for acquiring the current maximum block height of the block chain; setting an initial value of a current processing height, and acquiring a block from a block chain according to the current processing height and a block height flag bit; the block height flag bit comprises a first flag bit and a second flag bit of the corresponding height of each block on the block chain; the first flag bit is used for setting the corresponding height of the block and sending the block to a data filtering module; the second flag bit is used for setting the corresponding height of the block after the block processing is finished, and simultaneously adding one to the current processing height;
the data filtering module is used for checking the transaction records in the block, analyzing the required transaction records into structured data and sending the structured data to the data storage module;
and the data storage module is used for storing the structured data sent by the data filtering module.
9. The system according to claim 8, wherein the data detection module is further configured to obtain an unconfirmed transaction record from a memory pool of the blockchain and send the unconfirmed transaction record to the data filtering module; the data filtering module is also used for analyzing the needed unconfirmed transaction records into structured data and sending the structured data to the data storage module.
10. The system according to claim 8, wherein the data probing module is further configured to periodically check whether the hash value of the processed chunk is consistent with the hash value of the chunk with the corresponding height on the current blockchain, and if not, clear the second flag bit with the corresponding height in the chunk height flag bits.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911225763.5A CN111008218B (en) | 2019-12-04 | 2019-12-04 | Method and system for structured storage of data on block chain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911225763.5A CN111008218B (en) | 2019-12-04 | 2019-12-04 | Method and system for structured storage of data on block chain |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111008218A true CN111008218A (en) | 2020-04-14 |
| CN111008218B CN111008218B (en) | 2022-05-24 |
Family
ID=70113970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911225763.5A Active CN111008218B (en) | 2019-12-04 | 2019-12-04 | Method and system for structured storage of data on block chain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111008218B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559204A (en) * | 2020-12-08 | 2021-03-26 | 中山市当下科技有限公司 | Application data communication structure in UTXO account model block chain |
| CN113268546A (en) * | 2021-06-15 | 2021-08-17 | 中国电子科技网络信息安全有限公司 | Block chain account book data capture analysis method |
| CN113612761A (en) * | 2021-07-30 | 2021-11-05 | 立信(重庆)数据科技股份有限公司 | Block chain-based investigation data chaining method and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180039667A1 (en) * | 2016-08-05 | 2018-02-08 | Chicago Mercantile Exchange Inc. | Systems and methods for blockchain rule synchronization |
| CN108769173A (en) * | 2018-05-21 | 2018-11-06 | 阿里体育有限公司 | The block chain implementation method and equipment of the intelligent contract of operation |
| CN108805570A (en) * | 2018-06-01 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN109213900A (en) * | 2018-09-18 | 2019-01-15 | 百度在线网络技术（北京）有限公司 | A kind of data modification method, device, equipment and the medium of block chain |
| US20190116034A1 (en) * | 2017-10-12 | 2019-04-18 | Mastercard International Incorporated | Method and system for interacting public and private blockchains with controlled participation |
| CA2995177A1 (en) * | 2018-02-14 | 2019-08-14 | Beijing Tiande Technologies Limited | A method of state synchronization for complex smart contracts based on stage buckets |
| CN110309173A (en) * | 2019-06-14 | 2019-10-08 | 深圳前海达闼云端智能科技有限公司 | Contract data recording method and device, block chain node and storage medium |
| CN110430087A (en) * | 2019-09-16 | 2019-11-08 | 上海保险交易所股份有限公司 | The architecture design and realization of block chain heat upgrading |
-
2019
- 2019-12-04 CN CN201911225763.5A patent/CN111008218B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180039667A1 (en) * | 2016-08-05 | 2018-02-08 | Chicago Mercantile Exchange Inc. | Systems and methods for blockchain rule synchronization |
| US20190116034A1 (en) * | 2017-10-12 | 2019-04-18 | Mastercard International Incorporated | Method and system for interacting public and private blockchains with controlled participation |
| CA2995177A1 (en) * | 2018-02-14 | 2019-08-14 | Beijing Tiande Technologies Limited | A method of state synchronization for complex smart contracts based on stage buckets |
| CN108769173A (en) * | 2018-05-21 | 2018-11-06 | 阿里体育有限公司 | The block chain implementation method and equipment of the intelligent contract of operation |
| CN108805570A (en) * | 2018-06-01 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN109213900A (en) * | 2018-09-18 | 2019-01-15 | 百度在线网络技术（北京）有限公司 | A kind of data modification method, device, equipment and the medium of block chain |
| CN110309173A (en) * | 2019-06-14 | 2019-10-08 | 深圳前海达闼云端智能科技有限公司 | Contract data recording method and device, block chain node and storage medium |
| CN110430087A (en) * | 2019-09-16 | 2019-11-08 | 上海保险交易所股份有限公司 | The architecture design and realization of block chain heat upgrading |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559204A (en) * | 2020-12-08 | 2021-03-26 | 中山市当下科技有限公司 | Application data communication structure in UTXO account model block chain |
| CN113268546A (en) * | 2021-06-15 | 2021-08-17 | 中国电子科技网络信息安全有限公司 | Block chain account book data capture analysis method |
| CN113268546B (en) * | 2021-06-15 | 2022-04-01 | 中国电子科技网络信息安全有限公司 | Block chain account book data capture analysis method |
| CN113612761A (en) * | 2021-07-30 | 2021-11-05 | 立信(重庆)数据科技股份有限公司 | Block chain-based investigation data chaining method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111008218B (en) | 2022-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111008218B (en) | Method and system for structured storage of data on block chain | |
| CN108564470B (en) | Transaction distribution method for parallel building blocks in block chain | |
| CN101996102B (en) | Method and system for mining data association rule | |
| CN101043419B (en) | Method for an update synchronization of a domain information file and a requester data processing node | |
| CN103425734A (en) | Database, apparatus, and method for storing encoded triples | |
| CN107807932B (en) | Hierarchical data management method and system based on path enumeration | |
| CN105608135B (en) | Data mining method and system based on Apriori algorithm | |
| CN111651525A (en) | Block chain data storage method and device | |
| WO2022083436A1 (en) | Data processing method and apparatus, and device and readable storage medium | |
| CN105843809B (en) | Data processing method and device | |
| CN114816728A (en) | Elastic expansion method and system for cloud environment MongoDB database cluster instance node | |
| CN111046246B (en) | Label updating method and device and distributed storage system | |
| US20040044683A1 (en) | Data compiling method | |
| CN115129738B (en) | Cross-database data writing method, device and equipment | |
| CN111625617A (en) | Data indexing method and device and computer readable storage medium | |
| CN110795423A (en) | Data extraction method for rapid cleaning and conversion | |
| CN114969165A (en) | Data query request processing method, device, equipment and storage medium | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| EP3793171B1 (en) | Message processing method, apparatus, and system | |
| CN107515867A (en) | The generation method and device that data storage, querying method and the device and a kind of rowKey of a kind of NoSQL databases combine entirely | |
| CN110888863A (en) | State field optimization method and device, electronic equipment and storage medium | |
| CN111522805B (en) | Distributed batch data cleaning method and system | |
| CN111949686B (en) | Data processing method, device and equipment | |
| CN112711627B (en) | Data importing method, device and equipment of Greemplum database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |