CN112487027A - Efficient data query implementation method based on block chain electronic transaction - Google Patents
Efficient data query implementation method based on block chain electronic transaction Download PDFInfo
- Publication number
- CN112487027A CN112487027A CN202011390893.7A CN202011390893A CN112487027A CN 112487027 A CN112487027 A CN 112487027A CN 202011390893 A CN202011390893 A CN 202011390893A CN 112487027 A CN112487027 A CN 112487027A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- block chain
- metadata
- blockchain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a high-efficiency data query realization method based on block chain electronic transaction, which comprises the following steps: 1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing; 2) for the thermal data: supporting faster queries by building indexes or multi-node backups; 3) for cold data: and further compressing the data according to the stored database or carrying out few backups to achieve the aim of reducing the storage space. The invention starts the block chain system query optimization scheme from two aspects, firstly starts from storing a database, secondly establishes an efficient index mechanism, and stores the generated data into the database capable of providing rich query semantics through data monitoring, thereby improving the query efficiency, expanding the query function and enhancing the flexibility of data storage.
Description
Technical Field
The invention relates to a block chain electronic transaction-based efficient data query implementation method, and belongs to the technical field of big data retrieval.
Background
The block chain is originated from the bitcoin, and the trust problem in the transaction can be effectively solved through the technologies of a distributed account book, a consensus mechanism, an encryption algorithm and the like. In recent years, the development of block chains tends to be hot, commercial application projects are exploded, but the technology is not applied in a large scale, and only partial test is carried out on the aspects of finance, medical treatment, logistics and the like. There are still many problems in performance, safety, usability, and the technology thereof has not yet developed to maturity.
The block chain of fig. 1 is essentially a decentralized system, and is a chain data structure composed of a plurality of blocks. Each block is divided into a block head and a block body, the block head is mainly used for realizing the hash value of the block linked with the previous block, and the block body mainly comprises a transaction book.
Taking a transaction scenario as an example, the workflow is as follows:
(1) the client initiates a transaction, broadcasts the transaction to other nodes on the network after digital signature and waits for confirmation;
(2) the node in the network confirms and verifies the received transaction information, and after the verification is passed, the data is recorded into a block;
(3) and multiple receiving nodes in the whole network execute a consensus algorithm on the blocks, and the blocks are formally incorporated into a block chain for storage through the consensus algorithm.
Each user initiating a transaction in the transaction signs a randomly hashed digital signature to the previous transaction and the next owner's public key and attaches this signature to the end of the transferred electronic money, which is sent to the next owner, and the payee verifies the signature and can verify the owner of the chain, while each transaction is open, i.e. secured by a common recognition mechanism, in order to avoid double payment by the electronic money.
The blockchain query can be divided into account query, transaction query and contract query according to the query object. Although the block chain can support 3 types of queries, as the data size increases and the business requirements increase, the query disadvantage is gradually revealed, and the following points are mainly reflected.
Low efficiency of inquiry
As can be seen from the description of the levelDB, the levelDB is mainly suitable for scenes with more write operations and less read operations. The method comprises the steps of firstly accessing a memory, then accessing a cache, and finally sequentially querying SStables with different levels, wherein when the stored data volume is very large, the querying efficiency is very low. Moreover, with the ever-increasing data volume and the increasingly frequent query demands, it will become a major bottleneck of query performance.
② limited inquiry function
Most block chaining systems still use a levelDB-like key-value database, which is inherently high performing for large numbers of writes, but which only supports key-value based insertions and queries, and does not support lookups for any field in the value. Since support for analytic queries is not considered at the beginning of the design, no index is designed in the middle tier. The system cannot execute relational query like the traditional database, and cannot execute Top-K query, K-NN query and other complex analytical queries.
③ lack of flexibility in data storage
The number of fields of data stored in the current block chain is small, the structure is relatively fixed, and the query processing logic is simple. With the continuous expansion of the blockchain application, the data structure of the application is more complicated, but the current data storage has low storage expansibility for various data.
Disclosure of Invention
The technical task of the invention is to provide a high-efficiency data query implementation method based on block chain electronic transaction aiming at the defects of the prior art.
The technical scheme adopted by the invention for solving the technical problems is as follows:
the block chain electronic transaction-based efficient data query implementation method comprises the following steps:
1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing;
2) for the thermal data: supporting faster queries by building indexes or multi-node backups;
3) for cold data: and further compressing the block chain data according to a stored database or carrying out few backups to achieve the aim of reducing the storage space.
Further, step 2) for the hot data, optimizing query efficiency by adding an additional index into the level DB, taking the level DB as a main key index, establishing auxiliary indexes for different fields in the data storage module, and designing a query layer through a built-in index structure.
Further, the query process in step 2) is divided into two stages:
2.1) sending the query command to a query module, and determining a key (main key) set of results through the auxiliary index;
2.2) using the key value to find out a result value on a primary key index (levelDB) and returning the result value to the client.
Further, the built-in index structure design query layer in the step 2) comprises a block chain read/write API, a level db, a consistency maintenance module, a communication module and other nodes, the block chain read/write API is connected with the query module in the level db, the consistency maintenance module is connected with the other nodes through the communication module, and the consistency maintenance module is further connected with the auxiliary index module and the block chain read/write API.
Further, step 3) for cold data: and copying the block chain data into an external database, and designing a query layer by means of a functional interface provided by the external database.
Further, in step 3), for cold data, copying the block chain data into an external database by adopting an external database method, and designing a query layer by means of a functional interface provided by the external database.
Further, the etherQL system is adopted in the step 3), the data on the block chain is copied into the MongoDB, and then the analysis query operation is executed by using the Mongodb.
Further, the data processing in the step 1) comprises data extraction, data cleaning and data arrangement.
Further, in the step 1), the heterogeneous data sources include data in a block chain system and a decentralized distributed file system, metadata in the heterogeneous data sources are extracted and recorded into a metadata runtime library, the metadata are managed by using the metadata, the metadata are stored by using the metadata runtime library to form an analysis result for the data query module to query, and the metadata runtime library generates record description of the metadata on the data.
Compared with the prior art, the method for realizing efficient data query based on block chain electronic transaction has the advantages that,
the invention starts from two aspects, namely starting from a storage database and establishing an efficient index mechanism, monitors and stores the generated data into a database which can provide rich query semantics through data, then processes the stored data into cold and hot data, supports faster query by establishing indexes or multi-node backup for the hot data, and further compresses the cold data according to the stored database or performs less backup to achieve the purpose of reducing the storage space.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application.
FIG. 1 is a block diagram of a bit coin system according to the present invention;
FIG. 2 is a schematic diagram of the block chain-based electronic transaction query optimization method of the present invention;
FIG. 3 is a diagram of the structure of the built-in index method of the present invention.
Detailed Description
With reference to fig. 2, the invention discloses a method for realizing efficient data query based on block chain electronic transaction, which comprises the following steps:
1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing;
2) for the thermal data: supporting faster queries by building indexes or multi-node backups;
3) for cold data: and further compressing the block chain data according to a stored database or carrying out few backups to achieve the aim of reducing the storage space.
For step 1) above:
further, the data processing in the step 1) comprises data extraction, data cleaning and data arrangement.
Further, in the step 1), the heterogeneous data sources include data in a block chain system and a decentralized distributed file system, metadata in the heterogeneous data sources are extracted and recorded into a metadata runtime library, the metadata are managed by using the metadata, the metadata are stored by using the metadata runtime library to form an analysis result for the data query module to query, and the metadata runtime library generates record description of the metadata on the data.
For step 2) above:
further, step 2) for the hot data, optimizing query efficiency by adding an additional index into the level DB, taking the level DB as a main key index, establishing auxiliary indexes for different fields in the data storage module, and designing a query layer through a built-in index structure.
Further, the query process in step 2) is divided into two stages:
2.1) sending the query command to a query module, and determining a key (main key) set of results through the auxiliary index;
2.2) using the key value to find out a result value on a primary key index (levelDB) and returning the result value to the client.
With reference to fig. 3, further, the query layer of the built-in index structure design in step 2) includes a block chain read/write API, a level db, a consistency maintenance module, a communication module, and other nodes, where the block chain read/write API is connected to the query module in the level db, the consistency maintenance module is connected to the other nodes through the communication module, and the consistency maintenance module is further connected to the auxiliary index module and the block chain read/write API.
For step 3 above):
further, step 3) for cold data: and copying the block chain data into an external database, and designing a query layer by means of a functional interface provided by the external database.
Further, in step 3), for cold data, copying the block chain data into an external database by adopting an external database method, and designing a query layer by means of a functional interface provided by the external database.
Further, the etherQL system is adopted in the step 3), the data on the block chain is copied into the MongoDB, and then the analysis query operation is executed by using the Mongodb.
Claims (9)
1. The method for realizing efficient data query based on block chain electronic transaction is characterized by comprising the following steps of:
1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing;
2) for the thermal data: supporting faster queries by building indexes or multi-node backups;
3) for cold data: and further compressing the block chain data according to a stored database or carrying out few backups to achieve the aim of reducing the storage space.
2. The method as claimed in claim 1, wherein step 2) optimizes query efficiency for hot data by adding additional indexes into level db, using level db as main key index, building auxiliary indexes for different fields in the data storage module, and designing query layer through built-in index structure.
3. The method for implementing efficient data query based on blockchain electronic transaction as claimed in claim 2, wherein the query process of step 2) is divided into two stages:
2.1) sending the query command to a query module, and determining a key (main key) set of results through the auxiliary index;
2.2) using the key value to find out a result value on a primary key index (levelDB) and returning the result value to the client.
4. The method for implementing efficient data query based on blockchain electronic transaction according to claim 2, wherein the built-in index structure design query layer in step 2) includes a blockchain read/write API, a levelDB, a consistency maintenance module, a communication module, and other nodes, the blockchain read/write API is connected to the query module in the levelDB, the consistency maintenance module is connected to the other nodes through the communication module, and the consistency maintenance module is further connected to the auxiliary index module and the blockchain read/write API.
5. The method for implementing efficient data query based on blockchain electronic transaction as claimed in claim 1, wherein step 3) is implemented for cold data: and copying the block chain data into an external database, and designing a query layer by means of a functional interface provided by the external database.
6. The method for implementing efficient data query based on blockchain electronic transaction as claimed in claim 5, wherein step 3) uses an external database method for the cold data to copy the blockchain data into the external database, and designs the query layer by means of the functional interface provided by the external database.
7. The method as claimed in claim 3, wherein the etherQL system is used in step 3), the data on the block chain is copied to the MongoDB, and then the MongoDB is used to perform the analysis query operation.
8. The method as claimed in claim 1, wherein the data processing in step 1) includes data extraction, data cleaning, and data sorting.
9. The method for implementing efficient data query based on blockchain electronic transaction as claimed in claim 1, wherein the heterogeneous data sources in step 1) include data in a blockchain system and a decentralized distributed file system, metadata in the heterogeneous data sources is extracted and entered into a metadata runtime library, the metadata is managed by using the metadata, the metadata is stored by using the metadata runtime library to form an analysis result for query by the data query module, and the metadata runtime library generates record description of the metadata on the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011390893.7A CN112487027B (en) | 2020-12-02 | 2020-12-02 | Efficient data query implementation method based on block chain electronic transaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011390893.7A CN112487027B (en) | 2020-12-02 | 2020-12-02 | Efficient data query implementation method based on block chain electronic transaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112487027A true CN112487027A (en) | 2021-03-12 |
| CN112487027B CN112487027B (en) | 2022-08-23 |
Family
ID=74938851
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011390893.7A Active CN112487027B (en) | 2020-12-02 | 2020-12-02 | Efficient data query implementation method based on block chain electronic transaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112487027B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115794930A (en) * | 2023-02-08 | 2023-03-14 | 南京纯白矩阵科技有限公司 | Expandable multi-block chain data ETL processing system and method |
| CN118296083A (en) * | 2024-06-03 | 2024-07-05 | 中国科学院合肥物质科学研究院 | Distributed account book data three-level read-write expansion blockchain storage method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729371A (en) * | 2017-09-12 | 2018-02-23 | 深圳先进技术研究院 | The data directory and querying method of block chain, device, equipment and storage medium |
| CN109165224A (en) * | 2018-08-24 | 2019-01-08 | 东北大学 | A kind of indexing means being directed to keyword key on block chain database |
| CN109189782A (en) * | 2018-08-02 | 2019-01-11 | 哈尔滨工程大学 | A kind of indexing means in block chain commodity transaction inquiry |
| CN109347941A (en) * | 2018-10-10 | 2019-02-15 | 南京简诺特智能科技有限公司 | A kind of data sharing platform and its implementation based on block chain |
| CN109857722A (en) * | 2019-01-10 | 2019-06-07 | 厦门必乐领主科技有限公司 | A kind of data base array driving and dynamic index technology based on block chain |
| CN109885615A (en) * | 2019-01-24 | 2019-06-14 | 华东师范大学 | A kind of range query towards the light client of block chain based on index can verify that querying method |
| CN110020091A (en) * | 2018-10-16 | 2019-07-16 | 陕西医链区块链集团有限公司 | Medical search engine system based on block chain |
| CN110309196A (en) * | 2019-05-22 | 2019-10-08 | 深圳壹账通智能科技有限公司 | Block chain data storage and query method, apparatus, equipment and storage medium |
| CN111259056A (en) * | 2020-01-15 | 2020-06-09 | 深圳微众信用科技股份有限公司 | Block chain data query method, system and related equipment |
| CN111858520A (en) * | 2020-07-21 | 2020-10-30 | 杭州溪塔科技有限公司 | Method and device for separately storing block link point data |
-
2020
- 2020-12-02 CN CN202011390893.7A patent/CN112487027B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729371A (en) * | 2017-09-12 | 2018-02-23 | 深圳先进技术研究院 | The data directory and querying method of block chain, device, equipment and storage medium |
| CN109189782A (en) * | 2018-08-02 | 2019-01-11 | 哈尔滨工程大学 | A kind of indexing means in block chain commodity transaction inquiry |
| CN109165224A (en) * | 2018-08-24 | 2019-01-08 | 东北大学 | A kind of indexing means being directed to keyword key on block chain database |
| CN109347941A (en) * | 2018-10-10 | 2019-02-15 | 南京简诺特智能科技有限公司 | A kind of data sharing platform and its implementation based on block chain |
| CN110020091A (en) * | 2018-10-16 | 2019-07-16 | 陕西医链区块链集团有限公司 | Medical search engine system based on block chain |
| CN109857722A (en) * | 2019-01-10 | 2019-06-07 | 厦门必乐领主科技有限公司 | A kind of data base array driving and dynamic index technology based on block chain |
| CN109885615A (en) * | 2019-01-24 | 2019-06-14 | 华东师范大学 | A kind of range query towards the light client of block chain based on index can verify that querying method |
| CN110309196A (en) * | 2019-05-22 | 2019-10-08 | 深圳壹账通智能科技有限公司 | Block chain data storage and query method, apparatus, equipment and storage medium |
| CN111259056A (en) * | 2020-01-15 | 2020-06-09 | 深圳微众信用科技股份有限公司 | Block chain data query method, system and related equipment |
| CN111858520A (en) * | 2020-07-21 | 2020-10-30 | 杭州溪塔科技有限公司 | Method and device for separately storing block link point data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115794930A (en) * | 2023-02-08 | 2023-03-14 | 南京纯白矩阵科技有限公司 | Expandable multi-block chain data ETL processing system and method |
| CN118296083A (en) * | 2024-06-03 | 2024-07-05 | 中国科学院合肥物质科学研究院 | Distributed account book data three-level read-write expansion blockchain storage method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112487027B (en) | 2022-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220414090A1 (en) | Blockchain data index method, blockchain data storage method and device | |
| KR102392944B1 (en) | Data backup methods, storage media and computing devices | |
| CN108319654B (en) | Computing system, cold and hot data separation method and device, and computer readable storage medium | |
| CN111143389B (en) | Transaction execution method and device, computer equipment and storage medium | |
| US10649995B2 (en) | Methods and systems for optimizing queries in a multi-tenant store | |
| US11283616B2 (en) | Method for index-based and integrity-assured search in a blockchain | |
| US20190361913A1 (en) | Data replication technique in database management system | |
| US7584204B2 (en) | Fuzzy lookup table maintenance | |
| CN112487027B (en) | Efficient data query implementation method based on block chain electronic transaction | |
| CN105556520A (en) | Mirroring, in memory, data from disk to improve query performance | |
| CN105556519A (en) | Multi-version concurrency control on in-memory snapshot store of ORACLE in-memory database | |
| US11003540B2 (en) | Method, server, and computer readable medium for index recovery using index redo log | |
| US20100153346A1 (en) | Data integrity in a database environment through background synchronization | |
| WO2021179782A1 (en) | Method, device and apparatus for improving execution efficiency of database appliance, and medium | |
| CN102033930A (en) | Distributed memory database system | |
| CN106874399B (en) | Networking backup system and backup method | |
| CN110597835A (en) | Transaction data deleting method and device based on block chain | |
| CN115952195A (en) | Block chain data tracing query optimization method | |
| CN115599807A (en) | Data access method, device, application server and storage medium | |
| CN103501339A (en) | Metadata processing method and metadata server | |
| US7672925B2 (en) | Accelerating queries using temporary enumeration representation | |
| US20200019539A1 (en) | Efficient and light-weight indexing for massive blob/objects | |
| US20240095248A1 (en) | Data transfer in a computer-implemented database from a database extension layer | |
| US11726978B2 (en) | Computer program for providing efficient change data capture in a database system | |
| CN113204564A (en) | Database high-frequency SQL query method, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220802 Address after: 250100 building S02, No. 1036, Langchao Road, high tech Zone, Jinan City, Shandong Province Applicant after: Shandong Inspur Scientific Research Institute Co.,Ltd. Address before: 250100 First Floor of R&D Building 2877 Kehang Road, Sun Village Town, Jinan High-tech Zone, Shandong Province Applicant before: JINAN INSPUR HIGH-TECH TECHNOLOGY DEVELOPMENT Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |