CN115129790B

CN115129790B - Consistent decentralization storage system based on block chain

Info

Publication number: CN115129790B
Application number: CN202210894493.2A
Authority: CN
Inventors: 伍前红; 张品戈; 高启元; 程浩添
Original assignee: Beihang University
Current assignee: Beihang University
Priority date: 2022-07-28
Filing date: 2022-07-28
Publication date: 2024-04-19
Anticipated expiration: 2042-07-28
Also published as: CN115129790A

Abstract

The invention relates to a block chain based consistency decentralization storage system. The system comprises: the system comprises a blockchain interaction module, a data sharing module and a data persistence storage module. The data sharing module is connected with the block chain interaction module. The data persistence storage module is connected with the data sharing module. The block chain interaction module is used for acquiring a task type and generating a task scheduling instruction based on the task type; the data sharing module is used for opening up a storage space to store hot data and responding to a task scheduling instruction; the hot data is the data with the largest access times in a preset time period; and the data persistence storage module is used for unifying the hot data and the local storage data stored in the data sharing module to form k-v type data searched by using the key value pair so as to solve the problem of data island in the prior art.

Description

Consistent decentralization storage system based on block chain

Technical Field

The invention relates to the technical field of block chain data storage, in particular to a consistency decentralization storage system based on a block chain.

Background

At present, under the trend of mobile internetworking, the data volume of each industry presents a remarkable growth situation, and various information storage forms are generated at the same time. Overall, china is a world-wide data storage market in which global 2020 big data stores up to 47ZB are in rapid growth. The ever-expanding size of data storage drives economic development, but due to current architectural limitations, various potential problems continue to occur.

1. The problem of block chain data account book size limitation (the problem of state explosion, excessive storage space cost on contract interactive chains and the like caused by overlarge account volume of all nodes of the block chain)

Currently, with the continuous application of blockchain technology in various industries, data stored in blockchains is also becoming larger and larger. Taking the ethernet as an example, by day 28 of 5 months of 2022, the block snapshot storage of the ethernet full node has reached more than 10000GB and is also growing at a rate of 10GB per week. While some existing designs introduce a process of creating snapshots periodically and deleting old blocks that are not accessed after a period of time, already deployed systems such as bitcoin chains, ethernet chains, etc., cannot employ corresponding methods, a phenomenon known as state explosion of blockchains.

The ethernet chain proposes a corresponding node snapshot pruning method at the version 1.10.X client to reduce the storage space occupied by its state tree snapshot, but this method has many limitations, such as that its snapshot must be stored in a solid state disk, the archive node cannot be pruned, etc. Therefore, a method is lacking at present, and the synchronous time of the blockchain ledger is reduced under the condition that the requirement of the blockchain on equipment is not influenced, so that the problem of state explosion of the blockchain is solved.

2. Data consistency problem

In the process of accessing the database, the conditions of concurrency, multi-point access, data change and the like often occur, and the problems of error of data consistency, such as dirty data reading, unrepeatable reading and the like, are easy to occur. In order to ensure that access to the database in a period of time does not generate ambiguity and correct data is read, a shared log mode is adopted to ensure data consistency. A common table is designed in the shared log mechanism, namely, mapping is carried out on the whole database, the table can be accessed or modified at the same time, and consistency of the common table can be maintained, namely, consistency of the whole database can be maintained. Is also the most common method of constructing highly fault tolerant storage services.

However, the related data consistency ensuring method cannot well isolate the log and the physical storage, and meanwhile, the read-write speed may be reduced due to the corresponding increase of the log table, so that the efficiency problem is generated.

3. Data centralization problem

In the traditional cloud computing field, data storage mainly depends on centralized storage of each unit or enterprise, so that each data storage space generates each information island, and various problems, such as centralized storage of data, are extremely easy to attack and difficult to attack prevention, are caused; the data storage cost is high when the data is stored in the data set; the leakage risk of the data centralized storage is high, and the hazard of the data leakage is high; data storage is difficult to maintain and transmission speed is slow. Therefore, a method is needed to unify related various existing centralized storage systems, the information transmission barriers among the storage systems are opened, and the interconnection and the intercommunication of data are realized.

In this context, various types of distributed storage systems have been developed that utilize non-centralized storage to distribute data storage across storage nodes, thus correspondingly yielding various types of distributed storage systems, such as Napster, kazza, IPFS, etc. The system perfectly solves the data sharing requirement, has various characteristics of high downloading speed, safety, long data storage time and the like, and well solves various problems caused by centralized storage.

On a logic level, the distributed storage system effectively connects data scattered everywhere through a network communication technology by a plurality of storage nodes through a data transmission protocol, so that a distributed database can ensure that a user cannot feel the scattering of the data on a physical level in the using process, and the usability of the distributed storage system is greatly enhanced.

However, the related various distributed storage systems are less combined with the original data storage facilities, and meanwhile, due to the reasons of technical barriers, different protocols and the like, each centralized file storage system cannot join the network of the distributed storage system, so that the problem of data island is generated.

Disclosure of Invention

In order to solve the above problems in the prior art, the present invention provides a consistent de-centralized storage system based on blockchain.

In order to achieve the above object, the present invention provides the following solutions:

A blockchain-based consistent decentralized storage system, comprising:

The block chain interaction module is used for acquiring a task type and generating a task scheduling instruction based on the task type;

The data sharing module is connected with the block chain interaction module, and is used for opening up a storage space to store hot data and responding to the task scheduling instruction; the hot data is the data with the largest access times in a preset time period;

And the data persistence storage module is connected with the data sharing module and is used for unifying hot data and local storage data stored in the data sharing module to form k-v type data searched by utilizing key value pairs.

Preferably, the blockchain interaction module includes:

The data interaction interface is used for processing a writing request or a reading request of the blockchain verification state, collecting a data packet corresponding to the writing request or the reading request, analyzing the data packet to obtain a task type, and generating a global common identification read-write table based on the task type; the task type includes writing and reading; the global common identification read-write table is used for storing task types, task key names and task IDs;

the data consistency verification unit is connected with the data interaction interface and is used for protecting the data state storage by utilizing the global common identification read-write table to obtain a verification data packet;

and the task scheduling unit is connected with the data consistency verification unit and used for packaging the check data packet into an atomic task and scheduling the atomic task based on the characteristics of a task queue.

Preferably, the data consistency verification unit uses the global consensus read-write table to protect the data state storage to obtain a verification data packet, and specifically includes:

analyzing a task key name related to the task type according to the task type;

Acquiring a current task ID in the global common identification read-write table, determining whether a task key name of the current task is used, setting a corresponding process of the current task to be in a suspended state when the task key name of the current task is used by a task with a task ID smaller than the current task ID, and executing the current task when the task key name of the current task is not used by a task with a task ID smaller than the current task ID;

after the read-write task of the current task is blocked, broadcasting the read-write table, releasing the task blocking, and placing the released task into a message queue;

And when the current task is scheduled, updating the global common identification read-write table to finish the corresponding read-write task.

Preferably, the task scheduling unit schedules the atomic task based on the characteristics of a task queue, and specifically includes:

Acquiring the priority of the current task;

when the current task enters a task queue, sequencing the current task according to the priority.

Preferably, the data sharing module is formed by constructing an interstellar file system.

Preferably, the data sharing module includes:

The local unit is used as a node of the interstellar file system, is used for opening up a storage space for the file system by utilizing a key value in the interstellar file system to store hot data, and is used for responding to the task scheduling instruction;

and the cloud unit is used as a node of the interstellar file system, is used for storing the backup of the hot data and is used for responding to the task scheduling instruction.

Preferably, the data persistence storage module includes:

the universal abstract interface is used for receiving the task packaged by the blockchain interaction module and unifying databases of different types into k-v type data searched by utilizing key value pairs;

And the low-level database is connected with the general abstract interface and is used for storing the k-v type data.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

the invention provides a consistency decentralization storage system based on a block chain, which comprises the following components: the system comprises a blockchain interaction module, a data sharing module and a data persistence storage module. The data sharing module is connected with the block chain interaction module. The data persistence storage module is connected with the data sharing module. The block chain interaction module is used for acquiring a task type and generating a task scheduling instruction based on the task type; the data sharing module is used for opening up a storage space to store hot data and responding to a task scheduling instruction; the hot data is the data with the largest access times in a preset time period; and the data persistence storage module is used for unifying the hot data and the local storage data stored in the data sharing module to form k-v type data searched by using the key value pair so as to solve the problem of data island in the prior art.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a block chain based consistent de-centralized storage system architecture diagram provided by an embodiment of the present invention;

FIG. 2 is a block chain interaction module architecture diagram provided by an embodiment of the present invention;

FIG. 3 is a flow chart of read-write separation execution provided by an embodiment of the invention;

FIG. 4 is a read flow diagram of a blockchain-based consistent decentralized storage system according to an embodiment of the invention;

FIG. 5 is a write flow diagram of a blockchain-based consistent decentralized storage system according to an embodiment of the invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention aims to provide a block chain based consistency de-centering storage system, so as to solve the problem of block chain account book size limitation, and simultaneously, the data is shared by utilizing a IPFS storage mode while the consistency of related data in the calculation and verification process is ensured, so that the problem of data centering (namely data island) is solved.

In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.

As shown in FIG. 1, the blockchain-based consistent decentralized storage system provided by the invention can be divided into four parts, namely a blockchain interaction layer, a IPFS cache layer, a universal abstract interface and a bottom database from top to bottom according to functional division.

Then, the above-mentioned divided four physical layers of the present invention are divided into three functional modules, namely, a blockchain interaction module, a data sharing module and a data persistence storage module. The block chain interaction module corresponds to a block chain interaction layer in the physical layer, the data sharing module corresponds to a IPFS cache layer, and the data persistence storage module corresponds to a general abstract interface and a bottom database.

The following describes the implementation procedure of the three functional modules provided above:

A. block chain interaction module

The block chain interaction module mainly comprises a data interaction interface, a data consistency verification unit and a task scheduling unit, as shown in fig. 2.

The data interaction interface is used for processing a writing or reading request of a related blockchain verification state, collecting related data packets, analyzing the data packets, dividing input parameters into six parts including task types, key names, file values, hash values, database names and protection periods, and specifically, explaining the parameters in table 1.

Table 1 blockchain interactive module input parameter table

Instructions are divided into two types of reading and writing according to task types in table 1, wherein the writing instructions comprise modification, addition, deletion operations and the like for internal data, and the reading instructions mainly refer to inquiry for the internal data. In order to realize the consistency protection of state storage in the block chain verification and calculation process, the invention adopts a read-write separation architecture, respectively establishes a global consensus read-write table, separates the read-write structure, and the read-write table structure is shown in table 2.

Table 2 read-write table parameter design table

The data consistency verification unit protects the state storage by using the global consensus read-write table in table 2. Firstly, dividing tasks into two types of writing and reading according to the task types analyzed in the table 1, and simultaneously analyzing key names of related tasks. And the writing tasks and the reading tasks are divided into different flows according to the two types.

The write task is performed as follows:

1. And newly creating a task with each task type being written in a write table, and filling the task type and the key name of the task.

2. If the key name is used by a task with a task ID smaller than the current task ID in the read table or the write table, the corresponding process of the new task is set to be in a suspended state in order to protect the data consistency of the task which is currently using the data item.

3. And broadcasting the read-write table after the read-write task for blocking the task is completed, and releasing the task blocking.

4. The tasks that are released are placed in the message queue.

5. And the tasks are scheduled, the contents in the database are updated, and the corresponding writing tasks are completed. Through the writing flow, the aim of preventing different data sources from being read in the consistency verification process and failing to achieve the consistency verification is fulfilled.

The read task is performed as follows:

1. And newly creating a read task in the read list for each task type, and filling the task type and the key name.

2. Only checking whether the task ID in the write table is smaller than the task ID of the current task, if so, setting the corresponding process of the new task to be in a suspended state for protecting the data consistency of the task which uses the data item currently.

3. And broadcasting the read list after the task writing of the task is completed, and releasing the task blocking.

4. The tasks that are released are placed in the message queue.

5. And executing related reading tasks, and ensuring the reading consistency during the verification of the data source.

Based on the description, the data interaction interface provided by the invention can ensure read-write consistency without adding extra log storage, reduce extra use of storage and increase system operation efficiency.

In this embodiment, the overall flow of the data consistency verification unit is shown in fig. 3.

The task scheduling unit packages the data packets after the data consistency verification unit verifies the data packets into atomic tasks by utilizing the data structure of the task queue, and schedules access tasks by utilizing the characteristics of the task queue, when related tasks enter the task queue, the tasks are sequenced according to an algorithm set in a module, and the tasks with high priority are inserted into the front position in the queue, so that the tasks with higher related priority enter the system for processing, and the bottom layer memory is reasonably called, so that the system efficiency is highest as much as possible.

B. Data sharing module

The data sharing module is built by using an interstellar file system (IPFS), and is divided into a local unit and a cloud unit. The local unit opens up a relevant storage space for the file system by utilizing key values in IPFS, is used for storing hot data with the largest access times in a period of time in the database, carries out local caching on the hot data, and quickly responds to a data query task generated in the blockchain interaction module so as to accelerate the calling of relevant data.

Meanwhile, due to the nature of IPFS, data are put into a local unit for caching and are uploaded to the IPFS network cloud as shared data, the data are subjected to persistent backup, and other IPFS network nodes are shared, so that the problem of centralization of the data is solved. The local unit simultaneously joins the IPFS network as IPFS node and stores the related data of other nodes. The cloud unit comprises other nodes added into IPFS networks and is used for storing data backup and realizing data persistence.

C. Data persistence storage module

The data persistence storage module comprises a general abstract interface and an underlying database.

The universal abstract interface unifies the local IPFS cache layer with the local persistent storage: for a writing task, converting the operation of all databases into adding and deleting two types, and calling an adding interface corresponding to the data adding task; the data deleting task calls a deleting interface; a data modification task, calling an adding and deleting interface, and ensuring the atomicity of the adding and deleting task; for a read task, its query interface is invoked for all types of database operations. And receiving the task packaged by the related blockchain interaction module through the universal abstract interface, so that the databases of different types are unified into a k-v type database searched by using the key value pair.

The bottom database is used for storing persistent data and various data with larger volumes, such as various blockchain account book data, and is compatible with databases of different types and different storage states through the abstract interface, so that high universality and low coupling of the whole database connection are ensured. Meanwhile, through the data sharing module, the blockchain account data stored in the bottom database can be multiplexed, and the influence of blockchain account state explosion on the occupation of on-chain storage is reduced.

Through the description of the three modules, the whole system can efficiently complete the execution of two tasks of reading and writing under the condition of ensuring the consistency of data.

The flow of reading data is mainly divided into the following steps.

1. Firstly, analyzing the data packet, and obtaining a key corresponding to the data to be fetched according to the data items in the table 2.

2. After the analysis of the data packet is completed, consistency protection is performed for the searching process, whether corresponding data is being modified is searched, and if the corresponding data is being modified, the reading task is blocked.

3. And after the consistency protection is completed, the tasks are scheduled and distributed.

4. In the lookup process, the local IPFS cache is first looked up.

5. A miss, looking up a local persistent store.

6. And if the node is not hit, searching the storage of other IPFS nodes in the cloud. Because the whole system storage module is IPFS local cache, local persistent storage and IPFS cloud storage respectively from top to bottom. The method has the characteristic of gradual decrease in storage and reading speed, so that the high efficiency of system operation can be ensured during searching and accessing, meanwhile, the consistency protection of read files is increased, and the reliability of the system is enhanced. The overall read flow is shown in FIG. 4, where the solid lines represent the flow and the dotted horizontal lines represent the data.

Correspondingly, the data writing flow mainly comprises the following steps.

1. Firstly, analyzing the data packet, and obtaining keys and stored values corresponding to the required data according to the data items in the table 2.

2. And after the analysis of the data packet is completed, carrying out consistency protection on the data writing process.

4. It is looked up whether the file exists in the local persistent database before writing.

5. If not, the data is written into a local persistent database and IPFS cache, and meanwhile IPFS cache and a cloud IPFS node are synchronized. The overall write flow is shown in FIG. 5, where the solid lines represent the flow and the dotted horizontal lines represent the data.

Based on the above description, the present invention has the following advantages over the prior art:

1. realizing the sharing function of data while keeping the low coupling of the bottom database

The invention adopts a distributed database mode, utilizes a general abstract interface to share the data in the bottom layer persistent database, and simultaneously realizes low coupling of database connection. And the upper layer uses IPFS as storage to realize the caching function and sharing of data, so that the data reading speed is increased, and the problem of centralization of the data is solved by adding the sharing function.

The data can multiplex the blockchain account book in the bottom database for multiple times in the use process due to the data sharing mode, so that the on-chain storage use problem caused by the state explosion problem of the blockchain is reduced to a certain extent.

2. Ensuring consistency of data read-write in blockchain state verification

The invention respectively establishes the global common identification read-write table at the block chain interaction layer, separates the read-write, and ensures the data consistency of IPFS and the rear-end persistent database, thereby enhancing the robustness of data storage.

And when the read-write consistency is ensured, no additional log storage is established, the use of storage space is reduced, and the read-write efficiency is improved.

In summary, the invention aims to enhance the sharing property of stored data under the condition of ensuring the consistency of the data, open the communication barrier between data centers, ensure the low coupling property of interfaces and promote the universality of a data sharing system.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.

The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims

1. A blockchain-based consistent decentralized storage system, comprising:

the data persistence storage module is connected with the data sharing module and is used for unifying hot data and local storage data stored in the data sharing module to form k-v type data searched by utilizing key value pairs;

The blockchain interaction module includes:

The task scheduling unit is connected with the data consistency verification unit and used for packaging the verification data packet into an atomic task and scheduling the atomic task based on the characteristics of a task queue;

the data sharing module is formed by constructing an interstellar file system; the data sharing module includes:

the cloud unit is used as a node of the interstellar file system, is used for storing the backup of the hot data and is used for responding to the task scheduling instruction;

The data persistence storage module comprises:

2. The blockchain-based consistent decentralized storage system of claim 1, wherein the data consistency verification unit uses the global consensus read-write table to protect data state storage to obtain a verification data packet, and specifically comprises:

analyzing a task key name related to the task type according to the task type;

3. The blockchain-based consistent decentralized storage system according to claim 1, wherein the task scheduling unit schedules the atomic tasks based on characteristics of a task queue, specifically comprising:

Acquiring the priority of the current task;