CN113407512A - Block chain-based distributed storage system, method, device and storage medium - Google Patents
Block chain-based distributed storage system, method, device and storage medium Download PDFInfo
- Publication number
- CN113407512A CN113407512A CN202110647882.0A CN202110647882A CN113407512A CN 113407512 A CN113407512 A CN 113407512A CN 202110647882 A CN202110647882 A CN 202110647882A CN 113407512 A CN113407512 A CN 113407512A
- Authority
- CN
- China
- Prior art keywords
- data
- processing module
- block chain
- storage
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 145
- 238000013500 data storage Methods 0.000 claims abstract description 124
- 230000006870 function Effects 0.000 claims description 32
- 238000004590 computer program Methods 0.000 claims description 14
- 230000004048 modification Effects 0.000 claims description 14
- 238000012986 modification Methods 0.000 claims description 14
- 230000003993 interaction Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 239000008186 active pharmaceutical agent Substances 0.000 claims 1
- 238000012795 verification Methods 0.000 abstract 1
- 239000000306 component Substances 0.000 description 70
- 238000005516 engineering process Methods 0.000 description 11
- 238000012217 deletion Methods 0.000 description 8
- 230000037430 deletion Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- RTZKZFJDLAIYFH-UHFFFAOYSA-N Diethyl ether Chemical compound CCOCC RTZKZFJDLAIYFH-UHFFFAOYSA-N 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/50—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols using hash chains, e.g. blockchains or hash trees
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Storage Device Security (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed storage system, a distributed storage method, distributed storage equipment and a storage medium based on a block chain. The distributed storage system based on the block chain combines the block chain file system with the data storage assembly, the block chain file system is used for storing the data fingerprints and the data storage addresses, the data storage assembly is used for storing data, the validity verification can be carried out on the data stored in the data storage assembly through the data fingerprints and the data storage addresses stored in the block chain file system, and the data in the data storage assembly is prevented from being maliciously tampered. In addition, the block chain file system only needs to record the fingerprint, the address and the state of data, so that the storage pressure of the block chain system is reduced, and the processing efficiency of the block chain is improved. In addition, the data storage component is convenient to expand, can realize the storage support of various complex data, meets diversified storage requirements, and simultaneously realizes the large capacity of the whole storage.
Description
Technical Field
The present invention relates to the field of block chain technology, and in particular, to a distributed storage system and method based on a block chain, a device, and a computer-readable storage medium.
Background
The block chain technology is an emerging technology which is characterized in that a point-to-point distributed network is formed by a plurality of computer devices and a complete distributed database is maintained together. The block chain technology has the characteristics of decentralization, openness and transparency, difficulty in data tampering, difficulty in data loss and the like, and is widely applied to a plurality of fields.
However, the blockchain technology is strictly a decentralized distributed storage technology, and it uses P2P technology to perform networking communication, uses an encryption algorithm to prevent data tampering, and uses a consensus algorithm to agree on the storage contents of each node. As shown in fig. 1, the blockchain file system generally consists of two parts, namely, a chain file for recording specific events and a database for recording system status. Bitcoin, ether house, super book, etc. are all representatives of blockchain items, and although the encryption algorithm and the consensus mechanism used by them are different, they can generate trust and incentive, but are limited by the natural characteristics of blockchain technology, and at present, blockchain distributed storage has the following problems:
1) and low treatment efficiency: available blockchain items must support highly concurrent services, which puts high demands on the processing efficiency of blockchains, but according to the CAP principle, on the premise of ensuring consistency, it is difficult to achieve a great increase in efficiency.
2) And low storage capacity: the low storage capacity is a problem faced in the period of block chain length, and although the problem of single data storage content can be solved by adding a storage server in a federation chain, the solution can further reduce the processing efficiency of the block chain.
3) There is a conflict between data storage and actual demand: the blockchain usually stores transaction records in a data structure in a key-value pair form, does not support more complex file storage, such as pictures, audio, video and the like, and can store single content, while in an actual use scene, a plurality of services need more complex data as support, so that the ledger data cannot well meet actual service requirements, and the application expansion of the blockchain is limited.
Disclosure of Invention
The invention provides a distributed storage system based on a block chain, a method, equipment and a computer readable storage medium, which aim to solve the technical problems of low processing efficiency, low storage capacity and incapability of meeting diversified data storage requirements in the conventional distributed storage mode of the block chain.
According to an aspect of the present invention, there is provided a blockchain-based distributed storage system, including:
the system comprises a user side, a data processing system and a data processing system, wherein the user side is used for performing function operation on a user to generate a segment of data stream;
the server comprises a data processing module for correspondingly processing the data stream according to the function operation request of the user;
the storage terminal comprises a block chain file system and a data storage component, wherein the block chain file system is used for storing addresses and fingerprints of data and recording the state of the data, and the data storage component is used for storing the data;
the API gateway is used for aggregating the API of the block chain file system and the data storage component so as to be called by the data processing module;
and the data processing module receives the data stream, performs corresponding data processing according to different function operation requests of a user, calls a corresponding API (application program interface) to interact with the block chain file system and the storage assembly respectively, stores the data in the data storage assembly, stores the data fingerprint and the data storage address in the block chain file system, and responds to the user side according to an interaction result.
Further, the data storage component comprises at least one of a relational database for storing structured data, an NOSQL database for storing semi-structured data, a cache pool for storing data with high-speed read-write requirements, a decentralized distributed storage component for storing unstructured data and a centralized distributed storage component for storing unstructured data with files as minimum units, and the data processing module is further used for calling different APIs (application programming interfaces) to interact with different data storage components according to data types or data read-write requirements.
Further, when a user performs read-write operation, the data processing module performs fingerprint calculation on data generated by the read-write operation through a hash function, and stores the data into a corresponding data storage component according to the data type or the data read-write requirement, the data storage component sends a storage address to the data processing module, and the data processing module packs the data fingerprint and the data address and sends the data fingerprint and the data address to a block chain file system for uplink.
Further, when a user performs an inquiry operation, the data index is sent to the data processing module through the user side, the data processing module asks for a corresponding data fingerprint and a data address from the block chain file system by using the received data index, asks for data from the corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the block chain file system, the data is proved not to be tampered, and the data processing module returns the obtained data to the user side.
Further, when a user performs modification operation, the data index and modification content are sent to the data processing module through the user side, the data processing module utilizes the received data index to ask for a corresponding data fingerprint and a data address from the block chain file system and asks for data from the corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module utilizes a hash function to calculate the data fingerprint based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the block chain file system, the data is proved not to be tampered, the data processing module modifies the obtained data according to the received modification content, calculates a new data fingerprint of the modified data, and stores the modified data to the corresponding data storage component, and the data storage component sends the new data address to the data processing module, and the data processing module packs the new data fingerprint and the new data address and sends the packed new data fingerprint and the new data address to the block chain file system for uplink.
Further, when a user performs a deletion operation, a data index and a deletion instruction are sent to the data processing module through the user side, the data processing module uses the received data index to request a corresponding data fingerprint and a data address from the block chain file system, and sends the deletion instruction to the block chain file system, the block chain file system deletes the data fingerprint and the data address, the data processing module sends the obtained data address and the deletion instruction to the corresponding data storage component, and the data storage component deletes data.
Further, the server also comprises a plurality of reverse proxy modules for distributing the data streams, the number of the data processing modules is multiple, the reverse proxy modules can acquire the working state of each data processing module, and the reverse proxy modules distribute the data streams to the idle data processing modules after receiving the data streams.
In addition, the invention also provides a distributed storage method based on the block chain, which adopts the distributed storage system based on the block chain, and comprises the following contents:
the user performs functional operation to generate a segment of data stream;
correspondingly processing the data stream according to the function operation request of the user, calling a corresponding API (application program interface) to respectively interact with the block chain file system and the storage component, storing data in the data storage component, and storing the data fingerprint and the data storage address in the block chain file system;
and responding to the user side according to the interaction result.
In addition, the present invention also provides an apparatus comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the steps of the method by calling the computer program stored in the memory.
The present invention also provides a computer-readable storage medium for storing a computer program for distributed storage based on a block chain, where the computer program performs the steps of the method described above when the computer program runs on a computer.
The invention has the following effects:
according to the distributed storage system based on the block chain, after a user performs function operation to generate data, a data processing module performs corresponding processing on a data stream according to a function operation request of the user, wherein the processing comprises serialization of the data, data fingerprint calculation and data fingerprint ratio equalization, then a corresponding API is called to interact with a block chain file system and a storage component respectively, the data is stored in the data storage component, and the data fingerprint and a data storage address are stored in the block chain file system for uplink. By combining the blockchain file system with the data storage assembly, storing the data fingerprints and the data storage addresses by using the blockchain file system, and storing the data by using the data storage assembly, the validity of the data stored in the data storage assembly can be verified by using the data fingerprints and the data storage addresses stored in the blockchain file system, so that the data in the data storage assembly is prevented from being maliciously tampered. In addition, the block chain file system only needs to record the fingerprint, the address and the state of data, so that the storage pressure of the block chain system is reduced, and the processing efficiency of the block chain is improved. In addition, the data storage component is convenient to expand, can realize the storage support of various complex data, meets diversified storage requirements, and simultaneously realizes the large capacity of the whole storage.
In addition, the block chain-based distributed storage method, the block chain-based distributed storage device and the computer-readable storage medium have the advantages.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a schematic diagram of a conventional block chain file system.
Fig. 2 is a schematic diagram of a network architecture of a blockchain-based distributed storage system according to a preferred embodiment of the present invention.
Fig. 3 is a schematic diagram of a data processing process of the distributed storage system based on the blockchain in the preferred embodiment of the present invention when a user performs a read/write operation.
Fig. 4 is a schematic diagram of a data processing procedure of the block chain based distributed storage system in the preferred embodiment of the present invention when a user performs a query operation.
Fig. 5 is a schematic diagram of data processing procedure of the block chain based distributed storage system in the preferred embodiment of the present invention when a user performs a modification operation.
Fig. 6 is a schematic diagram of a data processing procedure of the block chain based distributed storage system when a user performs a delete operation in the preferred embodiment of the present invention.
Fig. 7 is a flowchart illustrating a distributed storage method based on a blockchain according to another embodiment of the present invention.
Detailed Description
The embodiments of the invention will be described in detail below with reference to the accompanying drawings, but the invention can be embodied in many different forms, which are defined and covered by the following description.
As shown in fig. 2, a preferred embodiment of the present invention provides a block chain based distributed storage system, including:
the system comprises a user side, a data processing system and a data processing system, wherein the user side is used for performing function operation on a user to generate a segment of data stream;
the server comprises a data processing module for correspondingly processing the data stream according to the function operation request of the user;
the storage terminal comprises a block chain file system and a data storage component, wherein the block chain file system is used for storing addresses and fingerprints of data and recording the state of the data, and the data storage component is used for storing the data;
the API gateway is used for aggregating the API of the block chain file system and the data storage component so as to be called by the data processing module;
and the data processing module receives the data stream, performs corresponding data processing according to different function operation requests of a user, calls a corresponding API (application program interface) to interact with the block chain file system and the storage assembly respectively, stores the data in the data storage assembly, stores the data fingerprint and the data storage address in the block chain file system, and responds to the user side according to an interaction result.
It can be understood that, in the distributed storage system based on the blockchain according to this embodiment, after a user performs a function operation to generate data, the data processing module performs corresponding processing on a data stream according to a function operation request of the user, where the processing includes serialization of the data, data fingerprint calculation, and data fingerprint comparison, and then calls a corresponding API to interact with the blockchain file system and the storage component respectively, store the data in the data storage component, and store the data fingerprint and the data storage address in the blockchain file system for uplink. By combining the blockchain file system with the data storage assembly, storing the data fingerprints and the data storage addresses by using the blockchain file system, and storing the data by using the data storage assembly, the validity of the data stored in the data storage assembly can be verified by using the data fingerprints and the data storage addresses stored in the blockchain file system, so that the data in the data storage assembly is prevented from being maliciously tampered. In addition, the block chain file system only needs to record the fingerprint, the address and the state of data, so that the storage pressure of the block chain system is reduced, and the processing efficiency of the block chain is improved. In addition, the data storage component is convenient to expand, can realize the storage support of various complex data, meets diversified storage requirements, and simultaneously realizes the large capacity of the whole storage.
It can be understood that the server further includes a plurality of reverse proxy modules for distributing the data stream, the number of the data processing modules is multiple, the reverse proxy modules can obtain the working state of each data processing module, and the reverse proxy modules distribute the data stream to the idle data processing modules after receiving the data stream. When a plurality of clients simultaneously generate data streams, in order to meet the requirement of high concurrency, the reverse proxy module can acquire the working state of each data processing module and distribute a plurality of data streams to each idle data processing module, so that load balancing of the front end is realized, and the high concurrency service scene is met.
It can be understood that the API is an interface exposed by the blockchain file system and the data storage component, which is convenient for use, and the API gateway aggregates these interfaces to implement functions of protocol conversion, request routing, fusing degradation, data caching, load balancing, and the like, and then uniformly exposes the interfaces to the data processing module in the service end for calling. The data processing module may call different APIs to interact with the blockchain file system and the data storage component according to different data streams (read and write requests).
In addition, the block chain file system and the data storage component belong to one part of a background storage end, the difference between the two parts is that the stored contents are different, and the data stream processed by the data processing module is finally sent to the block chain file system and the data storage component for storage. The block chain file system stores addresses and fingerprints of data and records states of the data, the data storage assembly is responsible for storage and backup of actual data, the data storage assembly feeds results back to the data processing module after corresponding operations are carried out, and interaction is carried out through the data processing module.
Each node in the blockchain system completely or partially has a blockchain file system, and the blockchain file system has three functions, namely, storing data addresses and data fingerprints, storing historical states and modification records of data, and preventing the data from being maliciously tampered. The data storage assembly comprises at least one of a relational database for storing structured data, an NOSQL database for storing semi-structured data, a cache pool for storing data with high-speed read-write requirements, a decentralized distributed storage assembly for storing unstructured data and a centralized distributed storage assembly for storing unstructured data with files as the minimum units, and the data processing module is further used for calling different APIs (application programming interfaces) to interact with different data storage assemblies according to data types or data read-write requirements. It can be understood that the relational database can be implemented by means of a relational database such as pgSQL and the like, and is built in a cluster manner; the NOSQL database is also built in a cluster mode; the cache pool is realized by a cache database such as Redis, and is also built in a cluster mode in order to ensure the performance of the cache pool; the decentralized distributed storage component is used for storing unstructured data such as audio and video and can be realized through IPFS, and the decentralized distributed storage component is used as a node in an IPFS private network existing depending on a block chain network; the centralized distributed storage component is realized through the HDFS, and the centralized distributed storage component is used as a node in an HDFS distributed storage system depending on a block chain network.
The distributed storage system based on the block chain can allow different types of storage components to be added, can provide good support for different types of data storage, and further provides diversified options for application development built on the distributed storage system. Meanwhile, different APIs can be called at the back end to interact with different data storage components according to data types or data reading and writing requirements, so that load balance of the storage end is achieved, and the requirements of high-concurrency service scenes can be well met. And, a part (data storage component) of the storage end is built in a cluster mode, and a part (block chain file system) is formed in a P2P network mode, and both have good expansibility. The expansibility of the conventional distributed storage system is greatly limited by hardware cost, the number of servers in the system is increased to pay high cost, but the expansibility of the distributed storage system is revealed to be the most striking nowadays when cloud service and container technology are developed to be mature. For a distributed storage system deployed in a cloud or a container, the number of servers or containers can be flexibly increased and decreased according to the needs of an actual production scene, so that the allocation of resources is optimized. The distributed storage system of the invention also takes node failure as a normal state, and adopts various effective error detection and fault recovery mechanisms, and the failure of some nodes can not cause data loss and can not influence the stability of the whole system, thus leading the system to be capable of stably and continuously providing service to the outside and ensuring the robustness and high fault tolerance of the system.
It can be understood that the block chain based distributed storage system of the invention combines the block chain technology with the excellent characteristics of other storage systems to form a new distributed data storage system, and finally builds a block chain based high-performance expandable distributed data platform to provide high available data support for applications built on the upper layer. The high-performance expandable distributed storage framework can be generally divided into four layers: the bottommost technical support layer provides various technical supports for the implementation of the storage system; the data storage layer is responsible for increasing, deleting, modifying and checking specific data; the data processing layer is responsible for data processing and interaction with each data storage component; the data platform on the uppermost layer provides various data interfaces for the application. The data platform needs to realize high availability, large capacity, high efficiency, expandability and strong adaptability, firstly has good support for high concurrent services, secondly solves the problems of low storage capacity and low processing efficiency of a block chain, thirdly ensures the expandability and stability of a storage system and fourthly supports various application development. The data processing layer is used as a core component of high-performance expandable distributed storage, the data processing module firstly realizes load balance and provides guarantee for high availability of a data platform, and the data processing module is also responsible for data processing including serialization of data, data fingerprint calculation, component interaction and the like. The data storage layer is composed of a blockchain file system and a plurality of selectable, high-performance and expandable data storage components, supports various types of data storage, and simultaneously ensures that stored data are not easily tampered. The technical support layer combines the technologies of Protocol Buffers, a P2P network, consistent hash, Gossip, gRPC, Merkle-DAG and the like together, and provides support for stable operation of each data storage component and even the whole storage system.
Specifically, as shown in fig. 3, when a user performs a read-write operation, the data processing module performs fingerprint calculation on data generated by the read-write operation through a hash function, and stores the data into a corresponding data storage component according to a data type or a data read-write requirement, where the data storage component sends a storage address to the data processing module, and the data processing module packages the data fingerprint and the data address and sends the data fingerprint and the data address to a block chain file system for uplink.
As shown in fig. 4, when a user performs an inquiry operation, a data index and an inquiry command are sent to a data processing module through the user side, the data processing module requests a corresponding data fingerprint and a data address from a blockchain file system by using the received data index, requests data from a corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the blockchain file system, it is proved that the data is not tampered, and the data processing module returns the obtained data to the user side.
As shown in fig. 5, when a user performs a modification operation, the data index and the modified content are sent to the data processing module through the user side, the data processing module uses the received data index to ask the corresponding data fingerprint and data address for the blockchain file system, and asks the corresponding data storage component for data according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint based on the obtained data by using a hash function, if the calculated data fingerprint is consistent with the data fingerprint obtained from the blockchain file system, it is proved that the data is not tampered, the data processing module modifies the obtained data according to the received modified content, calculates a new data fingerprint of the modified data, and stores the modified data in the corresponding data storage component, and the data storage component sends the new data address to the data processing module, and the data processing module packs the new data fingerprint and the new data address and sends the packed new data fingerprint and the new data address to the block chain file system for uplink.
As shown in fig. 6, when a user performs a deletion operation, a data index and a deletion instruction are sent to a data processing module through the user side, the data processing module requests a corresponding data fingerprint and data address from a blockchain file system by using the received data index, and sends the deletion instruction to the blockchain file system, the blockchain file system deletes the data fingerprint and the data address, the data processing module sends the obtained data address and the deletion instruction to a corresponding data storage component, and the data storage component deletes data.
It can be understood that when a user needs to operate on related data stored in the storage end, the data processing module needs to interact with the blockchain file system and the data storage component at the same time, so as to associate information stored in the blockchain file system with information stored in the data storage component, so that the information stored in the data storage component is verified effectively through the information stored in the blockchain file system, and the data stored in the data storage component is prevented from being tampered maliciously.
It is understood that, as shown in fig. 7, another embodiment of the present invention further provides a blockchain-based distributed storage method, preferably using the blockchain-based distributed storage system as described above, where the method includes the following steps:
step S1: the user performs functional operation to generate a segment of data stream;
step S2: correspondingly processing the data stream according to the function operation request of the user, calling a corresponding API (application program interface) to respectively interact with the block chain file system and the storage component, storing data in the data storage component, and storing the data fingerprint and the data storage address in the block chain file system;
step S3: and responding to the user side according to the interaction result.
It can be understood that, in the distributed storage method based on the blockchain according to this embodiment, after a user performs a function operation to generate data, the data processing module performs corresponding processing on a data stream according to a function operation request of the user, where the processing includes serialization of the data, data fingerprint calculation, and data fingerprint comparison, and then calls a corresponding API to interact with the blockchain file system and the storage component, respectively, store the data in the data storage component, and store the data fingerprint and the data storage address in the blockchain file system for uplink. By combining the blockchain file system with the data storage assembly, storing the data fingerprints and the data storage addresses by using the blockchain file system, and storing the data by using the data storage assembly, the validity of the data stored in the data storage assembly can be verified by using the data fingerprints and the data storage addresses stored in the blockchain file system, so that the data in the data storage assembly is prevented from being maliciously tampered. In addition, the block chain file system only needs to record the fingerprint, the address and the state of data, so that the storage pressure of the block chain system is reduced, and the processing efficiency of the block chain is improved. In addition, the data storage component is convenient to expand, can realize the storage support of various complex data, meets diversified storage requirements, and simultaneously realizes the large capacity of the whole storage.
When a user carries out read-write operation, the data processing module carries out fingerprint calculation on data generated by the read-write operation through a hash function and stores the data into a corresponding data storage component according to the data type or the data read-write requirement, the data storage component sends a storage address to the data processing module, and the data processing module packs the data fingerprint and the data address and sends the data fingerprint and the data address to a block chain file system for uplink.
When a user carries out query operation, a data index and a query instruction are sent to a data processing module through a user side, the data processing module asks for a corresponding data fingerprint and a data address from a block chain file system by using the received data index, asks for data from a corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the block chain file system, the data is proved not to be tampered, and the data processing module returns the obtained data to the user side.
When a user carries out modification operation, sending a data index and modification content to a data processing module through a user side, the data processing module asks for a corresponding data fingerprint and a data address from a block chain file system by using the received data index, asks for data from a corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the block chain file system, the data is proved not to be falsified, the data processing module modifies the obtained data according to the received modification content, calculates a new data fingerprint of the modified data, and stores the modified data to the corresponding data storage component, and the data storage component sends the new data address to the data processing module, and the data processing module packs the new data fingerprint and the new data address and sends the packed new data fingerprint and the new data address to the block chain file system for uplink.
When a user carries out deleting operation, a data index and a deleting instruction are sent to a data processing module through a user side, the data processing module utilizes the received data index to ask a corresponding data fingerprint and a corresponding data address for a block chain file system, the deleting instruction is sent to the block chain file system, the block chain file system deletes the data fingerprint and the data address, the data processing module sends the obtained data address and the deleting instruction to a corresponding data storage assembly, and the data storage assembly deletes data.
In addition, another embodiment of the present invention further provides an apparatus, which includes a processor and a memory, wherein the memory stores a computer program, and the processor is used for executing the steps of the method described above by calling the computer program stored in the memory.
In addition, another embodiment of the present invention further provides a computer readable storage medium for storing a computer program for distributed storage based on a block chain, where the computer program performs the steps of the method described above when the computer program runs on a computer.
The general form of computer readable media includes: floppy disk (floppy disk), flexible disk (flexible disk), hard disk, magnetic tape, any of its magnetic media, CD-ROM, any of the other optical media, punch cards (punch cards), paper tape (paper tape), any of the other physical media with patterns of holes, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), FLASH erasable programmable read only memory (FLASH-EPROM), any of the other memory chips or cartridges, or any of the other media from which a computer can read. The instructions may further be transmitted or received by a transmission medium. The term transmission medium may include any tangible or intangible medium that is operable to store, encode, or carry instructions for execution by the machine, and includes digital or analog communications signals or intangible medium that facilitates communication of the instructions. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a bus for transmitting a computer data signal.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A blockchain-based distributed storage system, comprising:
the system comprises a user side, a data processing system and a data processing system, wherein the user side is used for performing function operation on a user to generate a segment of data stream;
the server comprises a data processing module for correspondingly processing the data stream according to the function operation request of the user;
the storage terminal comprises a block chain file system and a data storage component, wherein the block chain file system is used for storing addresses and fingerprints of data and recording the state of the data, and the data storage component is used for storing the data;
the API gateway is used for aggregating the API of the block chain file system and the data storage component so as to be called by the data processing module;
and the data processing module receives the data stream, performs corresponding data processing according to different function operation requests of a user, calls a corresponding API (application program interface) to interact with the block chain file system and the storage assembly respectively, stores the data in the data storage assembly, stores the data fingerprint and the data storage address in the block chain file system, and responds to the user side according to an interaction result.
2. The blockchain-based distributed storage system of claim 1, wherein the data storage components include at least one of a relational database for storing structured data, an NOSQL database for storing semi-structured data, a cache pool for storing data with read-write-high requirements, a decentralized distributed storage component for storing unstructured data, and a centralized distributed storage component for storing unstructured data in a minimum unit of a file, and wherein the data processing module is further configured to invoke different APIs to interact with different data storage components according to data type or read-write requirements.
3. The blockchain-based distributed storage system of claim 1, wherein when a user performs a read/write operation, the data processing module performs fingerprint calculation on data generated by the read/write operation through a hash function and stores the data into a corresponding data storage component according to a data type or a data read/write requirement, the data storage component sends a storage address to the data processing module, and the data processing module packages the data fingerprint and the data address and sends the data fingerprint and the data address to a blockchain file system for uplink.
4. The blockchain-based distributed storage system of claim 1, wherein when a user performs an inquiry operation, the data index is sent to the data processing module through the user side, the data processing module requests a corresponding data fingerprint and data address from the blockchain file system by using the received data index, and requests data from a corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, and if the calculated data fingerprint is consistent with the data fingerprint obtained from the blockchain file system, the data processing module proves that the data is not tampered, and the data processing module returns the obtained data to the user side.
5. The blockchain-based distributed storage system of claim 1, wherein when a user performs a modification operation, the data processing module sends a data index and a modification content to the data processing module through the user side, the data processing module requests a corresponding data fingerprint and a data address from the blockchain file system by using the received data index, requests data from a corresponding data storage component according to the obtained data address, the data storage component sends the data to the data processing module, the data processing module calculates the data fingerprint by using a hash function based on the obtained data, if the calculated data fingerprint is consistent with the data fingerprint obtained from the blockchain file system, the data is proved not to be falsified, the data processing module modifies the obtained data according to the received modification content, and calculates a new fingerprint of the modified data, and storing the modified data in a corresponding data storage component, wherein the data storage component sends the new data address to a data processing module, and the data processing module packs the new data fingerprint and the new data address and sends the new data fingerprint and the new data address to a block chain file system for uplink.
6. The blockchain-based distributed storage system according to claim 1, wherein when a user performs a delete operation, a data index and a delete command are sent to the data processing module through the user side, the data processing module requests a corresponding data fingerprint and data address from the blockchain file system by using the received data index, and sends the delete command to the blockchain file system, the blockchain file system deletes the data fingerprint and the data address, the data processing module sends the obtained data address and the delete command to the corresponding data storage component, and the data storage component deletes data.
7. The blockchain-based distributed storage system according to claim 1, wherein the server further includes a plurality of reverse proxy modules for distributing the data stream, the number of the data processing modules is plural, the reverse proxy modules can obtain the working status of each data processing module, and the reverse proxy modules distribute the data stream to idle data processing modules after receiving the data stream.
8. A distributed storage method based on a block chain, which adopts the distributed storage system based on the block chain as claimed in any one of claims 1 to 7, and is characterized by comprising the following steps:
the user performs functional operation to generate a segment of data stream;
correspondingly processing the data stream according to the function operation request of the user, calling a corresponding API (application program interface) to respectively interact with the block chain file system and the storage component, storing data in the data storage component, and storing the data fingerprint and the data storage address in the block chain file system;
and responding to the user side according to the interaction result.
9. An apparatus, comprising a processor and a memory, the memory having stored therein a computer program, the processor being configured to perform the steps of the method of claim 8 by invoking the computer program stored in the memory.
10. A computer-readable storage medium for storing a computer program for distributed storage based on block chains, characterized in that the computer program performs the steps of the method as claimed in claim 8 when run on a computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110647882.0A CN113407512A (en) | 2021-06-10 | 2021-06-10 | Block chain-based distributed storage system, method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110647882.0A CN113407512A (en) | 2021-06-10 | 2021-06-10 | Block chain-based distributed storage system, method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113407512A true CN113407512A (en) | 2021-09-17 |
Family
ID=77683421
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110647882.0A Pending CN113407512A (en) | 2021-06-10 | 2021-06-10 | Block chain-based distributed storage system, method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113407512A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273410A (en) * | 2017-05-03 | 2017-10-20 | 上海点融信息科技有限责任公司 | Distributed storage based on block chain |
| CN109493055A (en) * | 2018-11-28 | 2019-03-19 | 深圳力维智联技术有限公司 | Networking block chain equipment, data processing method and device based on block chain |
| CN109981772A (en) * | 2019-03-22 | 2019-07-05 | 西安电子科技大学 | A kind of multiple domain data share exchange platform architecture based on block chain |
| CN110149353A (en) * | 2018-02-11 | 2019-08-20 | 陕西爱尚物联科技有限公司 | A kind of Internet of Things network method and its system |
| US20190311116A1 (en) * | 2016-12-06 | 2019-10-10 | Alibaba Group Holding Limited | Method, apparatus, and system for service data processing and verification |
| CN110912937A (en) * | 2019-12-23 | 2020-03-24 | 杭州中科先进技术研究院有限公司 | Block chain-based digital certificate storage platform and certificate storage method |
-
2021
- 2021-06-10 CN CN202110647882.0A patent/CN113407512A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190311116A1 (en) * | 2016-12-06 | 2019-10-10 | Alibaba Group Holding Limited | Method, apparatus, and system for service data processing and verification |
| CN107273410A (en) * | 2017-05-03 | 2017-10-20 | 上海点融信息科技有限责任公司 | Distributed storage based on block chain |
| CN110149353A (en) * | 2018-02-11 | 2019-08-20 | 陕西爱尚物联科技有限公司 | A kind of Internet of Things network method and its system |
| CN109493055A (en) * | 2018-11-28 | 2019-03-19 | 深圳力维智联技术有限公司 | Networking block chain equipment, data processing method and device based on block chain |
| CN109981772A (en) * | 2019-03-22 | 2019-07-05 | 西安电子科技大学 | A kind of multiple domain data share exchange platform architecture based on block chain |
| CN110912937A (en) * | 2019-12-23 | 2020-03-24 | 杭州中科先进技术研究院有限公司 | Block chain-based digital certificate storage platform and certificate storage method |
Non-Patent Citations (2)
| Title |
|---|
| 赵进延: "《数字政府 服务社会》", 31 December 2019, 知识产权出版社 * |
| 陈艳平，徐受蓉: "《Java语言程序设计实用教程 第2版》", 31 January 2019, 北京理工大学出版社 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11165667B2 (en) | Dynamic scaling of storage volumes for storage client file systems | |
| US20190068690A1 (en) | Automated management of resource attributes across network-based services | |
| CN110113420A (en) | Distributed Message Queue management system based on NVM | |
| CN106446159B (en) | A kind of method of storage file, the first virtual machine and name node | |
| US10609123B2 (en) | Hybrid quorum policies for durable consensus in distributed systems | |
| CN112910880B (en) | Virtual room creating method, system, device, equipment and medium | |
| CN108319623A (en) | A kind of fast resampling method, apparatus and data-base cluster | |
| CN108319618B (en) | Data distribution control method, system and device of distributed storage system | |
| CN105282244B (en) | a kind of data processing method, device, server and controller | |
| CN109547537A (en) | The method for realizing openstack High Availabitity based on SAN storage shared volume | |
| US11178197B2 (en) | Idempotent processing of data streams | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| WO2023284473A1 (en) | Data management method and apparatus, computer device, and storage medium | |
| CN108959313B (en) | Concurrent processing method and device for massive small files and storage medium | |
| WO2010086922A1 (en) | Storage system | |
| CN105978744B (en) | A kind of resource allocation methods, apparatus and system | |
| CN116414828A (en) | Data management method and related device | |
| CN113885798A (en) | Data operation method, device, equipment and medium | |
| CN110618790A (en) | Mist storage data redundancy removing method based on repeated data deletion | |
| CN113472864B (en) | High-performance block chain distributed storage system, method, equipment and storage medium | |
| CN113407512A (en) | Block chain-based distributed storage system, method, device and storage medium | |
| CN109522294A (en) | A kind of distributed data cache system and data cache method | |
| CN116701452A (en) | Data processing method, related device, storage medium and program product | |
| CN105389368A (en) | Method for managing metadata of database cluster of MPP architecture | |
| CN112579343A (en) | Block link point data recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210917 |
|
| RJ01 | Rejection of invention patent application after publication |