CN111949630B - Distributed block chain storage method based on file review mechanism - Google Patents
Distributed block chain storage method based on file review mechanism Download PDFInfo
- Publication number
- CN111949630B CN111949630B CN202010794779.4A CN202010794779A CN111949630B CN 111949630 B CN111949630 B CN 111949630B CN 202010794779 A CN202010794779 A CN 202010794779A CN 111949630 B CN111949630 B CN 111949630B
- Authority
- CN
- China
- Prior art keywords
- file
- user
- storage
- super node
- examination
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1834—Distributed file systems implemented based on peer-to-peer networks, e.g. gnutella
- G06F16/1837—Management specially adapted to peer-to-peer storage networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of distributed storage, and provides a distributed block chain storage method based on a file review mechanism, which comprises the following steps: step 100, constructing a distributed block chain storage system architecture, wherein the storage system architecture comprises user nodes, super nodes, storage nodes and a cloud service platform; 200, acquiring files uploaded by a user and checking real-name authentication information of the user; step 300, the super node gateway of the super node executes a pre-examination mechanism to automatically examine the validity of the file, and performs distributed file storage on the pre-examination legal file; step 400, the super node gateway of the super node executes the post-verification mechanism to verify the validity of the file. The invention can contain the characteristics of distributed storage and high throughput when the file storage system is constructed, and simultaneously ensure that data can be audited and effectively supervised.
Description
Technical Field
The invention relates to the technical field of distributed storage, in particular to a distributed block chain storage method based on a file review mechanism.
Background
With the development of blockchain technology and interplanetary file system technology, more and more enterprises begin to build distributed file storage schemes, and distributed file storage schemes with complete decentralization and no supervision review are generally adopted at present.
But such a completely decentralized storage presents a great safety risk. Lawbreakers use the anonymity and decentralized characteristics of the distributed file system to spread illegal files to the distributed network, and the system cannot check and delete files in a targeted manner, so that crime risks exist.
Disclosure of Invention
The invention mainly solves the technical problems that the completely decentralized storage in the prior art has larger potential safety hazard and cannot carry out comprehensive data examination and management, and provides a distributed block chain storage method based on a file examination mechanism, which can contain the characteristics and high throughput of distributed storage and simultaneously ensure that data can be examined and effectively supervised when a file storage system is built.
The distributed block chain storage method based on the file review mechanism provided by the invention constructs a complete distributed file storage network which is respectively a user node, a super node, a storage node and a cloud service platform, and can obtain a robust, stable and data-neutralized distributed storage environment. The user node is responsible for initiating storage requirements to the network or downloading required files through the P2P network; the super node is responsible for the distribution and examination of the network storage order; the storage node is responsible for storing files and needs to guarantee the authenticity of its storage. The implementation of the file review mechanism is mainly completed by a super node in the network architecture. The real-name super nodes have high system performance, file metadata information is stored in each super node, search requests are only forwarded among the super nodes, and the super nodes can forward query requests to appropriate storage nodes. The super node service comprises a gateway of the P2P network and allows a user to upload files for payment, and the super node is responsible for receiving and processing a storage order of the user and distributing file fragments to be stored to the storage nodes in a fair manner. When finding that the file is not suitable for propagation, the gateway service of the super node needs to ask all storage nodes to delete the file, add the hash of the file into a blacklist, and disallow the hash to be returned as a retrieval result. The blacklisted file list is generally proposed to the super node after the relevant organization audits the file. Although the sharing of documents in the blacklist requires the super nodes in the network to achieve the common knowledge, the super nodes with the real name can easily achieve the common knowledge.
In the uploading process of the file, the method comprehensively examines the file through two parts of pre-examination and post-examination. The pre-examination mechanism is automatically completed by an AI document examination model served by the super node gateway, the document after the pre-examination passes is subjected to manual examination of the post-examination mechanism, and if the document still meets the requirement, the document is formally determined as a safety document. The invention can contain the characteristics of distributed storage and high throughput when the file storage system is constructed, and simultaneously ensure that data can be audited and effectively supervised.
Drawings
FIG. 1 is a flowchart of an implementation of a distributed blockchain storage method based on a file review mechanism according to the present invention;
FIG. 2 is a connection block diagram of a storage system architecture;
FIG. 3 is a flowchart of an implementation of step 300;
FIG. 4 is a flowchart of an implementation of step 400;
fig. 5 is a flowchart of an implementation of step 500.
Detailed Description
In order to make the technical problems solved, technical solutions adopted and technical effects achieved by the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings.
Fig. 1 is a flowchart of an implementation of a distributed blockchain storage method based on a file review mechanism according to an embodiment of the present invention. As shown in fig. 1, the distributed blockchain storage method based on a file review mechanism according to an embodiment of the present invention includes the following steps:
step 100, constructing a distributed block chain storage system architecture, wherein the storage system architecture comprises a user node 1, a super node 2, a storage node 3 and a cloud service platform 4.
FIG. 2 is a connection block diagram of a storage system architecture. As shown in fig. 2, the user node 1 is a system entry for uploading and downloading files by a user, and is responsible for auditing user real-name authentication information.
And the super node 2 stores the file metadata by using the block chain and stores the file in a distributed storage node network 3 in blocks. The super node 2 mainly realizes file examination and file storage management functions. Each super node 2 comprises a super node gateway, a file block chain and a file blacklist. The super node gateway comprises an AI model capable of being dynamically upgraded and is used for realizing intelligent examination of the file; and simultaneously, the system comprises a file transmission management function and is used for transmitting the suspicious files to the cloud server for manual examination. And the file block chain is used for recording information such as file names, file headers and the like to realize file query management. And the file blacklist is used for collecting the file names of the illegal files uploaded to the file system and shielding the inquiry of the illegal files on the block chain.
Each distributed storage node 3 comprises a file block storage and a file hash table; the file hash table is used for storing hash value information of all file blocks of the node to realize file block position index.
And the cloud service platform 4 is used for assisting in realizing the functions of node authority authentication and file examination and realizing a high-efficiency safe file examination mechanism. The cloud service platform 4 completes user authentication on the auxiliary user node 1; and performing authentication and authorization on the super node 2. An AI file examination algorithm library capable of being dynamically upgraded is established in the cloud service platform 4 by combining various artificial intelligence algorithms, so that the functions of image, video, character recognition, data classification and the like are realized, and the accuracy and the coverage of file examination are improved. The cloud service platform 4 provides a file manual inspection workstation, receives suspicious files transmitted by the super node gateway, and relevant personnel perform manual inspection on the files at the workstation. And the AI file algorithm library updates the file characteristics of the illegal files which are learned and manually checked by the workstation, and pushes the AI model with wider coverage to the super node gateway.
And 200, utilizing the files uploaded by the acquisition user and being responsible for checking the real-name authentication information of the user. Step 200 comprises steps 201-202:
in step 201, the user node 1 obtains a file storage request initiated by a user.
Step 202, the user node 1 verifies whether the user is a real-name user; if the user is not the real-name user, the file storage request is rejected and the operation is finished; if the user is a real-name user, step 300 will be entered.
And step 300, the super node gateway of the super node 2 executes a pre-examination mechanism to automatically examine the validity of the file, and performs distributed file storage on the pre-examination legal file. FIG. 3 is a flowchart of an implementation of step 300; as shown in FIG. 3, step 300 includes steps 301-303:
step 301, the super node gateway of the super node 2 executes a pre-verification mechanism to automatically check the validity of the file.
The file is uploaded to the super node 2, a gateway service of the super node 2 executes a file pre-examination mechanism, the file pre-examination mechanism conducts intelligent file examination on the file through a latest AI model pushed from an AI algorithm library of the cloud service platform 4, and the intelligent file examination is achieved through technologies such as multimedia information identification and data similarity analysis. In the process of identifying illegal files, if the files are previous-examination illegal files, rejecting the file storage request; if the file is a front-check legal file, step 302 is entered. It should be noted that the AI document verification model in the supernode gateway service may be dynamically updated to increase the flexibility of document verification.
Step 302, the file is agreed by super node 2 and the file summary information is stored in the block chain.
Step 303, the super node 2 sends the file fragments to the storage node 3, and performs distributed file storage based on the file fragmentation technology and the peer-to-peer file transfer protocol.
Step 400, the super node gateway of the super node 2 checks the validity of the document by the checking mechanism after executing.
After the file is stored in the storage node 3, the file security examination is not finished formally, and the file needs to be examined again through a post-examination mechanism. FIG. 4 is a flowchart of an implementation of step 400; as shown in FIG. 4, step 400 includes steps 401-402:
step 401, in a post-examination mechanism, the file will be transmitted to a file examination workstation of the cloud service platform 4 by the super node, and relevant personnel will manually examine the file; if the file is judged to be a post-examination legal file through manual examination, the file is stored and formally listed as a safety file; if the file is judged to be an unqualified file after manual examination, the super node 2 controls the storage node 3 to delete all the fragmented file blocks, adds the hash value of the file in the block chain into a file blacklist record for shielding, and simultaneously informs a user that the file is deleted and stored unsuccessfully.
Step 402, the cloud service platform 4 updates the AI document examination algorithm library according to the new illegal documents, and the cloud service platform 4 pushes the new models trained through the AI algorithm library to each super node gateway, so that the illegal documents of the same type are identified and filtered in the process of a pre-examination mechanism when the documents are uploaded, and the document examination efficiency is optimized.
In the method, the file is comprehensively inspected through two parts of pre-inspection in the step 300 and post-inspection in the step 400 in the uploading process. The pre-examination mechanism is automatically completed by an AI document examination model served by the super node gateway, the document after the pre-examination passes is subjected to manual examination of the post-examination mechanism, and if the document still meets the requirement, the document is formally determined as a safety document.
Step 500, the user initiates a file downloading request to complete the file downloading.
FIG. 5 is a flowchart of an implementation of step 500; as shown in FIG. 5, the detailed process of step 500 includes steps 501-505.
Step 501, a user initiates a file downloading request to a super node 2 through a user node 1.
Step 502, the download request is received by the super node 2, and whether the summary information of the required file is included is inquired in the block chain; if the file is not recorded in the block chain, the downloading request is refuted; if the required file exists, go to step 503.
Step 503, the super node 2 searches a file blacklist, and inquires whether the hash value of the file required by the user exists in the file blacklist; if the file exists in the blacklist, the downloading request is refuted; if the file is a legal file, step 504 is entered.
Step 504, the super node 2 retrieves the file fragment positions where the files are distributed in the IPFS storage network, and instructs the storage node 3 set capable of forming complete files and having the highest transmission efficiency to transmit the files to the user node 1 through the P2P network.
And 505, the user node receives the file fragments sent by the storage node 3 set and integrates the file fragments into a complete file, so as to complete file downloading.
The invention can ensure that when a user initiates a downloading request, the distributed high-speed P2P file downloading is realized, and meanwhile, the illegal file is prevented from being spread in a network.
The distributed block chain storage method based on the file review mechanism provided by this embodiment provides a function of using a block chain for a user by using the user node 1. The interface layer realizes data fragmentation in the process of uploading data and aims to uniformly divide the data into different storage servers; storing user metadata and storing files in a distributed storage node network 3 in blocks by using the super node 2; the storage node 3 is utilized to realize distributed storage of data and storage management of the data, and the functions of content storage, distribution and the like provided by IPFS are adopted; the cloud service platform 4 is utilized to assist in realizing the functions of node authority authentication and file examination, and a high-efficiency and safe file examination mechanism is realized. In summary, the scheme can ensure that when a user initiates a download request, distributed high-speed P2P file download is realized, and meanwhile, it is ensured that illegal files cannot be spread in a network.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: modifications of the technical solutions described in the embodiments or equivalent replacements of some or all technical features may be made without departing from the scope of the technical solutions of the embodiments of the present invention.
Claims (3)
1. A distributed block chain storage method based on a file review mechanism is characterized by comprising the following steps:
step 100, constructing a distributed block chain storage system architecture, wherein the storage system architecture comprises user nodes, super nodes, storage nodes and a cloud service platform;
200, acquiring files uploaded by a user and checking real-name authentication information of the user;
step 300, the super node gateway of the super node executes a pre-examination mechanism to automatically examine the validity of the file, and performs distributed file storage on the pre-examination legal file; step 300 includes steps 301-303:
step 301, the super node gateway of the super node automatically checks the validity of the file by performing a pre-verification mechanism: uploading the file to a super node, executing a file pre-examination mechanism by a gateway service of the super node, and intelligently examining the file by a latest AI model pushed from an AI algorithm library of a cloud service platform by the pre-examination mechanism; in the process of identifying illegal files, if the files are previous-examination illegal files, rejecting the file storage request; if the file is a front-check legal file, go to step 302;
step 302, the file is agreed by the super node, and the file abstract information is stored in a block chain;
step 303, the super node sends the file fragments to storage nodes, and performs distributed file storage based on a file fragmentation technology and a peer-to-peer file transfer protocol;
step 400, the super node gateway of the super node checks the legality of the file by a checking mechanism after executing; step 400 includes steps 401-402:
step 401, in a post-examination mechanism, the file is transmitted to a file examination workstation of the cloud service platform by the super node, and relevant personnel perform manual examination on the file; if the file is judged to be a post-examination legal file through manual examination, the file is stored and formally listed as a safety file; if the file is judged to be an illegal file after being checked and verified manually, the super node controls the storage node to delete all the fragmented file blocks, adds the hash value of the file corresponding to the block chain into a file blacklist record for shielding, and simultaneously informs a user that the file is deleted and stored unsuccessfully;
step 402, the cloud service platform updates an AI document examination algorithm library according to the new illegal documents, and the cloud service platform pushes the new models trained by the AI algorithm library to each super node gateway, so that the illegal documents of the same type are identified and filtered in the process of a pre-examination mechanism when the documents are uploaded.
2. The distributed blockchain storage method based on file review mechanism as claimed in claim 1, wherein step 200 includes the steps of:
step 201, a user node acquires a file storage request initiated by a user;
step 202, a user node verifies whether the user is a real-name user; if the user is not the real-name user, the file storage request is rejected and the operation is finished; if the user is a real-name user, step 300 will be entered.
3. The distributed blockchain storage method based on file review mechanism according to claim 1 or 2, further comprising:
step 500, a user initiates a file downloading request to complete file downloading; step 500 includes steps 501-505;
step 501, a user initiates a file downloading request to a super node through a user node;
step 502, a download request is received by a super node, and whether the summary information of a required file is contained or not is inquired in a block chain; if the file is not recorded in the block chain, the downloading request is refuted; if the required file exists, go to step 503;
step 503, the super node searches a file blacklist, and queries whether the hash value of the file required by the user exists in the file blacklist; if the file exists in the blacklist, the downloading request is refuted; if the file is a legal file, go to step 504;
step 504, the super node retrieves the file distribution in the file fragment position in the IPFS storage network, and commands can form a complete file and simultaneously transmit the file to the user node through the P2P network by the storage node set with the highest transmission efficiency;
and 505, the user node receives the file segments sent by the storage node set and integrates the file segments into a complete file, and the file downloading is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010794779.4A CN111949630B (en) | 2020-08-10 | 2020-08-10 | Distributed block chain storage method based on file review mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010794779.4A CN111949630B (en) | 2020-08-10 | 2020-08-10 | Distributed block chain storage method based on file review mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111949630A CN111949630A (en) | 2020-11-17 |
| CN111949630B true CN111949630B (en) | 2022-11-01 |
Family
ID=73333075
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010794779.4A Active CN111949630B (en) | 2020-08-10 | 2020-08-10 | Distributed block chain storage method based on file review mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949630B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114598693B (en) * | 2020-12-07 | 2023-11-21 | 国家广播电视总局广播电视科学研究院 | File content auditing method and device and electronic equipment |
| CN114006717A (en) * | 2021-01-04 | 2022-02-01 | 北京八分量信息科技有限公司 | Block chain link point storage cloud system |
| CN114221767A (en) * | 2021-11-02 | 2022-03-22 | 北京中合谷投资有限公司 | File monitoring method for distributed network |
| CN114221956A (en) * | 2021-11-08 | 2022-03-22 | 北京中合谷投资有限公司 | Content examination method of distributed network |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107249046A (en) * | 2017-08-15 | 2017-10-13 | 李俊庄 | A kind of distributed cloud storage system construction method based on block chain |
| CN109783466A (en) * | 2018-12-25 | 2019-05-21 | 上海点对点科技有限公司 | A kind of document management apparatus and method of the distributed storage network based on block chain |
| CN110399338B (en) * | 2019-04-13 | 2022-03-08 | 西安电子科技大学 | Distributed file index system and method based on block chain and cloud storage server |
| CN110427346A (en) * | 2019-08-13 | 2019-11-08 | 河南中盾云安信息科技有限公司 | A kind of document storage system and its file memory method based on block chain |
| CN110647503A (en) * | 2019-10-09 | 2020-01-03 | 重庆特斯联智慧科技股份有限公司 | Distributed storage method and device |
| CN110855760A (en) * | 2019-10-29 | 2020-02-28 | 上海海加网络科技有限公司 | Block chain-based distributed secure storage system deployment method |
-
2020
- 2020-08-10 CN CN202010794779.4A patent/CN111949630B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111949630A (en) | 2020-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111949630B (en) | Distributed block chain storage method based on file review mechanism | |
| CN109246211B (en) | Resource uploading and resource requesting method in block chain | |
| US8010666B2 (en) | Identification and tracking of digital content distributors on wide area networks | |
| KR101868029B1 (en) | Method and system for sharing file based on blockchain | |
| US8924460B2 (en) | Method and system of administrating a peer-to-peer file sharing network | |
| US8103870B2 (en) | Hive-based peer-to-peer network | |
| KR20060057563A (en) | The system and method that fragments data into unit and transmits it between client and server | |
| US20100211608A1 (en) | Apparatus and method for generating a database that maps metadata to p2p content | |
| CN108429759A (en) | Decentralization stores safety implementation method | |
| US20040181688A1 (en) | Systems and methods for the copy-protected distribution of electronic documents | |
| US20050091167A1 (en) | Interdiction of unauthorized copying in a decentralized network | |
| Norvill et al. | IPFS for reduction of chain size in Ethereum | |
| CN105321108A (en) | System and method for creating a list of shared information on a peer-to-peer network | |
| KR20080113227A (en) | Method and communication system for the computer-aided detection and identification of copyrighted contents | |
| US20220269754A1 (en) | Decentralized system and method for asset registry and authentication | |
| CN113065961A (en) | Power block chain data management system | |
| US20100223472A1 (en) | Method and system for legally sharing files | |
| WO2020196981A1 (en) | Method and system for preventing distribution of illegal content over the internet | |
| US20100191756A1 (en) | Content delivery apparatus, content delivery method, and content delivery program | |
| CN113378218B (en) | Intellectual property data storage and authentication method based on block chain | |
| CN113179328A (en) | Resource distribution method and system for PCDN (Primary Contourlet distribution) network | |
| US20190372882A1 (en) | Communication device and communication method | |
| CN108449348B (en) | Online authentication system and method supporting user identity privacy protection | |
| CN111353177A (en) | Block chain-based privacy large file storage system | |
| CN105893511A (en) | Method for data copy trace retention through agent cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |