CN115129790A

CN115129790A - Consistency decentralized storage system based on block chain

Info

Publication number: CN115129790A
Application number: CN202210894493.2A
Authority: CN
Inventors: 伍前红; 张品戈; 高启元; 程浩添
Original assignee: Beihang University
Current assignee: Beihang University
Priority date: 2022-07-28
Filing date: 2022-07-28
Publication date: 2022-09-30
Anticipated expiration: 2042-07-28
Also published as: CN115129790B

Abstract

The invention relates to a block chain-based consistent decentralized storage system. The system comprises: the device comprises a block chain interaction module, a data sharing module and a data persistence storage module. The data sharing module is connected with the block chain interaction module. The data persistence storage module is connected with the data sharing module. The block chain interaction module is used for acquiring the task type and generating a task scheduling instruction based on the task type; the data sharing module is used for opening up a storage space to store hot data and responding to a task scheduling instruction; the hot data is the data with the maximum access times in a preset time period; the data persistence storage module is used for unifying hot data and local storage data stored in the data sharing module to form k-v type data searched by using key values, so as to solve the problem of data isolated island in the prior art.

Description

Consistency decentralized storage system based on block chain

Technical Field

The invention relates to the technical field of block chain data storage, in particular to a block chain-based consistency decentralized storage system.

Background

At present, under the trend of mobile internet, the data volume of each industry presents a remarkable growth situation, and various information storage forms are generated at the same time. In general, china and even the world are in a fast growing stage of data storage market size, with the worldwide 2020 large data storage volume reaching 47 ZB. The ever-expanding scale of data storage has pushed economic development, but due to current architectural limitations, various potential problems continue to emerge.

1. The problem of limitation of the size of a block chain data account book (the problem of too high cost of storage space on a state explosion and contract interaction chain due to too large volume of the account book of all nodes of the block chain)

At present, with the continuous application of the blockchain technology in various industries, the data stored in the blockchain is larger and larger. Taking etherhouse as an example, by 28 days 5 and 28 months 2022, the block snapshot storage of the etherhouse full nodes has reached more than 10000GB and is also growing at a rate of 10GB per week. While some existing designs introduce a process of creating snapshots periodically and deleting old blocks that are not accessed after a period of time, deployed systems such as bitcoin chains, etherhouse chains, etc. cannot adopt a corresponding method, a phenomenon known as state explosion of a block chain.

The ethernet shop chain proposes a corresponding node snapshot pruning method at a 1.10.x version client, which is used for reducing the storage space occupied by the state tree snapshot, but the method also has many limitations, such as that the snapshot must be stored in a solid state disk, the archive node cannot be pruned, and the like. Therefore, a method is lacked at present, and under the condition that the requirement of the block chain for equipment is not influenced, the synchronization time of the block chain account book is reduced, so that the problem of state explosion of the block chain is solved.

2. Data consistency problem

In the database access process, situations such as concurrency, multi-point access, data change and the like often occur, and problems of data consistency errors are easily generated, such as dirty data reading, unrewritable data and the like. In order to ensure that access to the database does not generate ambiguity and correct data is read within a period of time, currently, a log sharing mode is mostly adopted to ensure data consistency. A common identification table is designed in a shared log mechanism, namely mapping of the whole database, the common identification table can be accessed or modified simultaneously, and the consistency of the common identification table is maintained, namely the consistency of the common identification table to the whole database is maintained. Is also the most common method of building highly fault tolerant storage services.

However, the method for ensuring consistency of related data cannot well isolate logs and physical storage, and simultaneously, due to the fact that a log table is correspondingly added, the reading and writing speed is possibly reduced, and therefore the efficiency problem is caused.

3. Data centralization problem

In the traditional cloud computing field, data storage mainly depends on centralized storage of each unit or enterprise, so that each data storage space generates respective information isolated islands, and various problems are caused, such as data centralized storage, easy attack and difficult attack prevention; the storage cost of data centralized storage is high; the data centralized storage has high leakage risk and high data leakage hazard; data storage is difficult to maintain and slow to transfer. Therefore, a method is needed for unifying various related existing centralized storage systems, getting through information transmission barriers between the storage systems, and realizing interconnection and intercommunication of data.

With this as a background, various distributed storage systems have been developed, and data storage is distributed on each storage node by using decentralized storage, so that various distributed storage systems such as Napster, Kazza, IPFS, and the like are correspondingly developed. The system perfectly meets the requirement of data sharing, has the characteristics of high downloading speed, safety, long data storage time and the like, and better solves various problems caused by centralized storage.

In the distributed storage system, a plurality of storage nodes effectively connect data scattered at all places through a network communication technology on a logic level through a data transmission protocol, so that the distributed database can ensure that a user cannot feel the scattering of the data on a physical level in the using process, and the usability of the distributed database is greatly enhanced.

However, various related distributed storage systems are rarely combined with the original data storage facilities, and meanwhile, due to the reasons of technical barriers, different protocols and the like, each centralized file storage system cannot be added into a network of the distributed storage system, so that the problem of data islanding is caused.

Disclosure of Invention

To solve the above problems in the prior art, the present invention provides a block chain based consistent decentralized storage system.

In order to achieve the purpose, the invention provides the following scheme:

a block chain based coherent decentralized storage system comprising:

the block chain interaction module is used for acquiring a task type and generating a task scheduling instruction based on the task type;

the data sharing module is connected with the block chain interaction module, is used for opening up a storage space to store hot data and is used for responding to the task scheduling instruction; the hot data is data with the maximum number of access times in a preset time period;

and the data persistence storage module is connected with the data sharing module and is used for unifying hot data and local storage data stored in the data sharing module to form k-v type data searched by using key value pairs.

Preferably, the block chain interaction module includes:

the data interaction interface is used for processing a write request or a read request of a block chain verification state, collecting a data packet corresponding to the write request or the read request, analyzing the data packet to obtain a task type, and generating a global consensus read-write table based on the task type; the task type comprises writing and reading; the global consensus read-write table is used for storing task types, task key names and task IDs;

the data consistency verification unit is connected with the data interaction interface and used for protecting data state storage by utilizing the global consensus read-write table to obtain a check data packet;

and the task scheduling unit is connected with the data consistency verification unit, and is used for encapsulating the check data packet into an atomic task and scheduling the atomic task based on the characteristics of a task queue.

Preferably, the data consistency verification unit uses the global consensus read-write table to protect data state storage to obtain a check data packet, and specifically includes:

analyzing a task key name related to the task type according to the task type;

acquiring a current task ID in the global consensus read-write table, determining whether a task key name of the current task is used, setting a corresponding process of the current task to be in a suspended state when the task key name of the current task is used by a task of which the task ID is smaller than the current task ID, and executing the current task when the task key name of the current task is not used by a task of which the task ID is smaller than the current task ID;

after the read-write task blocking the current task is completed, broadcasting the read-write table, releasing the task blocking, and putting the released task into a message queue;

and when the current task is scheduled, updating the global consensus read-write table to complete the corresponding read-write task.

Preferably, the task scheduling unit schedules the atomic task based on the characteristics of the task queue, and specifically includes:

acquiring the priority of the current task;

and when the current task enters a task queue, sequencing the current task according to the priority.

Preferably, the data sharing module is built by using an interplanetary file system.

Preferably, the data sharing module includes:

the local unit is used as a node of the interplanetary file system, is used for opening up a storage space for the file system to store hot data by using a key value in the interplanetary file system, and is used for responding to the task scheduling instruction;

and the cloud end unit is used as a node of the interplanetary file system, is used for storing backups of the hot data and is used for responding to the task scheduling instruction.

Preferably, the data persistence storage module comprises:

the universal abstract interface is used for receiving the tasks packed by the block chain interaction module and unifying different types of databases into k-v type data searched by using key value pairs;

and the low-layer database is connected with the universal abstract interface and is used for storing the k-v type data.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

the invention provides a block chain-based consistency decentralized storage system, which comprises: the device comprises a block chain interaction module, a data sharing module and a data persistence storage module. The data sharing module is connected with the block chain interaction module. The data persistence storage module is connected with the data sharing module. The block chain interaction module is used for acquiring the task type and generating a task scheduling instruction based on the task type; the data sharing module is used for opening up a storage space to store hot data and responding to a task scheduling instruction; the hot data is the data with the maximum access times in a preset time period; the data persistence storage module is used for unifying hot data and local storage data stored in the data sharing module to form k-v type data searched by using key values, so as to solve the problem of data isolated island in the prior art.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a block chain based consistent decentralized memory system architecture diagram according to an embodiment of the present invention;

fig. 2 is a block chain interactive module architecture diagram according to an embodiment of the present invention;

FIG. 3 is a flow chart of read/write separation according to an embodiment of the present invention;

FIG. 4 is a read flow diagram of a block chain based coherent decentralized memory system according to an embodiment of the present invention;

FIG. 5 is a write flow diagram of a block chain based coherent decentralized storage system according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention aims to provide a block chain-based consistency decentralized storage system, which solves the problem of size limitation of a block chain account book, ensures the consistency of related data in a calculation and verification process, shares the data in an IPFS (internet protocol file system) storage mode and solves the problem of data centralization (namely data islanding).

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.

As shown in fig. 1, the block chain-based coherent decentralized storage system provided by the present invention can be divided into four parts, from top to bottom, a block chain interaction layer, an IPFS cache layer, a generic abstraction interface, and a bottom database according to functional division.

Next, the four physical layers divided in the invention are divided into three functional modules, namely a block chain interaction module, a data sharing module and a data persistence storage module. The block chain interaction module corresponds to a block chain interaction layer in the physical layer, the data sharing module corresponds to an IPFS cache layer, and the data persistence storage module corresponds to a general abstract interface and a bottom database.

The following describes the specific implementation process of the three functional modules provided above:

A. block chain interaction module

The block chain interaction module mainly comprises a data interaction interface, a data consistency verification unit and a task scheduling unit, as shown in fig. 2.

The data interaction interface is used for processing a related block chain verification state writing or reading request, collecting related data packets, analyzing the data packets, and dividing input parameters into six parts, namely a task type, a key name, a file value, a hash value, a database name and a protection period, wherein the specific parameter explanation is shown in table 1.

Table 1 input parameter table for blockchain interaction module

The instructions are divided into two types of reading and writing according to the task types in table 1, wherein the writing instructions include operations of modification, addition, deletion and the like on internal data, and the reading instructions mainly refer to query on the internal data. In order to realize the consistency protection of state storage in the verification and calculation process of the block chain, the invention adopts a read-write separation architecture, respectively establishes globally identified read-write tables, and separates the read-write structure, wherein the structure of the read-write table is shown in table 2.

TABLE 2 design table of reading and writing table parameters

The data consistency verification unit utilizes the global consensus read-write table in the table 2 to protect the state storage. Firstly, the tasks are divided into a writing type and a reading type according to the task types analyzed in the table 1, and the key names of the related tasks are analyzed at the same time. And is divided into different flows according to the two types of writing and reading tasks.

The write task is performed as follows:

1. and newly creating a task with each task type as writing in the writing table, and filling the task type and the key name of the task.

2. If the key name is being used by the task whose task ID is smaller than the current task ID in the reading or writing list, the corresponding process of the newly-built task is set to be in a suspended state in order to protect the data consistency of the task which is using the data item.

3. And after the read-write task blocking the task is completed, broadcasting the read-write table and releasing the task block.

4. The associated released task is placed in a message queue.

5. And scheduling the tasks, updating the contents in the database and completing the corresponding writing tasks. Through the writing process, the situation that different data sources are read in the consistency verification process is prevented, and the purpose of consistency verification is not achieved.

The read task is performed as follows:

1. and newly creating a read table for each task type which is the read task, and filling the task type and the key name.

2. And only checking whether the task ID smaller than the current task ID is used in the write table, if so, setting the corresponding process of the newly-built task to be in a suspended state in order to protect the data consistency of the task using the data item currently.

3. And after the writing task blocking the task is completed, broadcasting the reading table and releasing the task block.

4. The associated released task is placed in a message queue.

5. And executing related reading tasks to ensure the reading consistency during the verification of the data source.

Based on the above description, the data interaction interface provided by the invention can ensure the read-write consistency without adding extra log storage, reduce the extra use of storage and increase the system operation efficiency.

In this embodiment, the overall flow of the data consistency verification unit is as shown in fig. 3.

The task scheduling unit encapsulates the data packets verified by the data consistency verification unit into atomic tasks by using a data structure of the task queue, and schedules the access tasks by using the characteristics of the task queue, when the related tasks enter the task queue, the tasks are sequenced according to an algorithm set in the module, and the tasks with high priority are inserted into the front position in the queue, so that the tasks with higher related priority enter the system to be processed first, and the underlying memory is reasonably called, so that the system efficiency is as highest as possible.

B. Data sharing module

The data sharing module is built by utilizing an interplanetary file system (IPFS), and is divided into a local unit and a cloud unit. The local unit opens up a related storage space for the file system by using the key value in the IPFS, is used for storing hot data with the maximum access frequency in a period of time in the database, performs local cache on the hot data, quickly responds to a data query task generated in the block chain interaction module, and accelerates the calling of the related data.

Meanwhile, due to the nature of the IPFS, the data can be uploaded to the IPFS network cloud as shared data while being cached in the local unit, persistent backup is carried out on the data, and other IPFS network nodes are shared, so that the problem of centralization of the data is solved. And the local unit is simultaneously used as an IPFS node to join the IPFS network and store the related data of other nodes. The cloud unit comprises other nodes added into the IPFS network and is used for storing data backup and realizing data persistence.

C. Data persistence storage module

The data persistence storage module comprises a universal abstract interface and an underlying database.

The universal abstract interface unifies the local IPFS cache layer with the local persistent storage: for writing tasks, converting all database operations into two types of adding and deleting, adding tasks corresponding to data, and calling an adding interface; a data deleting task calls a deleting interface; the data modification task calls an adding and deleting interface and ensures the atomicity of the adding and deleting task; for the read task, for all types of database operations, its query interface is invoked. And receiving tasks packed by the related block chain interaction module through the universal abstract interface, so that different types of databases are unified into a k-v type database searched by using key value pairs.

The bottom-layer database is used for storing persistent data and various types of data with large volumes, such as various types of block chain account book data, and the high universality and the low coupling of the connection of the whole database are ensured by the compatibility of the databases with different types and different storage states through the abstract interface. Meanwhile, through the data sharing module, block chain account book data stored in a bottom database can be multiplexed, and the influence of block chain account book state explosion on chain storage occupation is reduced.

Through the description of the three modules, the whole system can efficiently complete the execution of the two types of read-write tasks under the condition of ensuring the data consistency.

The flow of reading data is mainly divided into the following steps.

1. Firstly, the data packet is analyzed, and the key corresponding to the data needing to be taken out is obtained corresponding to the data item in the table 2.

2. After the data packet is analyzed, consistency protection is carried out on the searching process, whether the corresponding data is modified or not is searched, and if the corresponding data is modified, the reading task is blocked.

3. And after the consistency protection is finished, scheduling and distributing the tasks.

4. In the search process, the local IPFS cache is first searched.

5. On a miss, the local persistent store is looked up.

6. And if not, searching the storage of other IPFS nodes in the cloud. The storage module of the whole system is respectively an IPFS local cache, a local persistent storage and an IPFS cloud storage from top to bottom. The storage and reading speed has the characteristic of sequential decrement, so that the high efficiency of system operation can be ensured during searching and accessing, meanwhile, the consistency protection of read files is increased, and the system reliability is enhanced. The overall read flow is shown in fig. 4, where solid lines represent flow and dotted lines represent data.

Correspondingly, the data writing process is mainly divided into the following steps.

1. The data packet is first parsed to obtain the key corresponding to the data needed and the stored value corresponding to the data items in table 2.

2. And after the data packet analysis is completed, carrying out consistency protection on the data writing process.

4. The local persistent database is looked up for the presence of the file before writing.

5. And if the IPFS node does not exist, writing the IPFS node into the local persistent database and the IPFS cache, and synchronizing the IPFS cache and the cloud IPFS node. The overall write flow is shown in fig. 5, where solid lines represent flow and dotted horizontal lines represent data.

Based on the above description, the present invention has the following advantages over the prior art:

1. the low coupling of a bottom database is kept, and meanwhile, the data sharing function is realized

The invention adopts a distributed database mode, utilizes a general abstract interface to share data in a bottom-layer persistent database, and simultaneously realizes low coupling of database connection. And the upper layer utilizes IPFS as storage, so that the caching function and sharing of data are realized, the data reading speed is accelerated, and meanwhile, the problem of centralization of data is solved by adding a sharing function.

Due to the adoption of a data sharing mode, the block chain account book in the bottom database can be reused for multiple times in the using process, so that the problem of on-chain storage and use caused by the state explosion problem of the block chain is reduced to a certain extent.

2. Ensuring consistency of data read and write in block chain state verification

According to the method, the global consensus read-write table is respectively established on the block chain interaction layer, read-write is separated, and data consistency is guaranteed for the IPFS and the rear-end persistent database, so that the robustness of data storage is enhanced.

When the read-write consistency is ensured, no additional log storage is established, the use of storage space is reduced, and the read-write efficiency is improved.

In summary, the present invention is directed to enhancing the sharing performance of stored data, getting through the communication barrier between data centers, simultaneously ensuring the low coupling performance of interfaces, and improving the versatility of the data sharing system under the condition of ensuring the data consistency.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the foregoing, the description is not to be taken in a limiting sense.

Claims

1. A blockchain-based coherent decentralized storage system, comprising:

2. The blockchain-based coherent decentralized storage system according to claim 1, wherein said blockchain interaction module comprises:

the data consistency verification unit is connected with the data interaction interface and used for protecting data state storage by utilizing the global consensus read-write table to obtain a verification data packet;

and the task scheduling unit is connected with the data consistency verification unit, and is used for packaging the check data packet into an atomic task and scheduling the atomic task based on the characteristics of a task queue.

3. The block chain-based consistency decentralized storage system according to claim 2, wherein the data consistency verification unit uses the global consensus read-write table to protect data state storage to obtain a check data packet, and specifically includes:

analyzing a task key name related to the task type according to the task type;

4. The block chain-based coherent decentralized storage system according to claim 2, wherein the task scheduling unit schedules the atomic task based on a characteristic of a task queue, and specifically comprises:

acquiring the priority of the current task;

5. The blockchain-based coherent decentralized storage system according to claim 2, wherein said data sharing module is built using an interplanetary file system.

6. The blockchain-based coherent decentralized storage system according to claim 5, wherein said data sharing module comprises:

the local unit is used as a node of the interplanetary file system, is used for opening up a storage space for the file system by using a key value in the interplanetary file system to store hot data, and is used for responding to the task scheduling instruction;

7. The blockchain-based coherent decentralized storage system according to claim 1, wherein said data persistence storage module comprises:

and the low-level database is connected with the universal abstract interface and is used for storing the k-v type data.