CN111309699A

CN111309699A - Content sharing method and system based on point-to-point distributed file system

Info

Publication number: CN111309699A
Application number: CN202010043517.4A
Authority: CN
Inventors: 张文龙
Original assignee: Individual
Current assignee: Individual
Priority date: 2020-01-15
Filing date: 2020-01-15
Publication date: 2020-06-19

Abstract

The invention relates to a content sharing method and a system based on a point-to-point distributed file system, wherein the method comprises the following steps: when receiving a file uploaded by a user through a sharing node, storing the uploaded file in a storage node in a point-to-point distributed file system where the sharing node is located; when a reading request initiated by a reading node for a file stored in the point-to-point distributed file system is received, initiating a charging request to the reading node according to the cost corresponding to the file requested to be read; and when the reading node is confirmed to complete fee payment according to the charging request, distributing the paid fee to accounts corresponding to the sharing node and the storage node according to a preset rule, and caching the file requested to be read to the reading node. The invention can encourage users to provide storage resources for users in need by using idle disks, and is convenient for the users to distribute contents, so that the users and the users can benefit at the same time.

Description

Content sharing method and system based on point-to-point distributed file system

Technical Field

The invention relates to the field of file sharing, in particular to a content sharing method and system based on a point-to-point distributed file system.

Background

In the current knowledge payment era, a chance of obtaining income by means of self knowledge is brought to the majority of creators, but at present, a plurality of problems exist, on one hand, a certain early investment is needed for the creators to distribute files containing knowledge contents through an internet channel, on the other hand, the huge file amount also brings a severe test for the storage of the files, and the problem of high storage cost is brought by only depending on the storage space provided by a service provider.

Disclosure of Invention

In view of the above technical problems, the present invention provides a content sharing method and system based on a peer-to-peer distributed file system.

The technical scheme for solving the technical problems is as follows: a content sharing method based on a point-to-point distributed file system comprises the following steps:

when receiving a file uploaded by a user through a sharing node, storing the uploaded file in a storage node in a point-to-point distributed file system where the sharing node is located;

when a reading request initiated by a reading node for a file stored in the point-to-point distributed file system is received, initiating a charging request to the reading node according to the cost corresponding to the file requested to be read;

and when the reading node is confirmed to complete fee payment according to the charging request, distributing the paid fee to accounts corresponding to the sharing node and the storage node according to a preset rule, and caching the file requested to be read to the reading node.

The invention has the beneficial effects that: the user who has original resource issue can issue own original content and price through the sharing node, the user who has idle storage resource can contribute part of the hard disk of the user, the user who has demand can pay for purchase after seeing the content that the user wants through reading the node, the received cost is returned to the person who issues the resource and the person who provides the storage resource according to a certain proportion, thereby encouraging the user to offer the disk that has idle to provide the storage resource for the user who has demand, and the user is convenient for content distribution, so that the two parties benefit simultaneously.

On the basis of the technical scheme, the invention can be further improved as follows.

Further, the storing the uploaded file in a storage node in a point-to-point distributed file system where the sharing node is located specifically includes:

the method comprises the steps of segmenting an uploaded file into a plurality of file blocks, calculating the hash value of each file block, storing the file blocks in storage nodes in a point-to-point distributed file system where sharing nodes are located through a p2p peer-to-peer network, and establishing a DHT (distributed hash table) according to the corresponding relation between the hash values of the file blocks and the storage nodes.

Further, caching the file requested to be read to the reading node specifically includes:

and searching a corresponding storage node from the DHT according to the file block hash value of the file requested to be read, reading a file block from the searched storage node and caching the file block to the reading node.

Further, when receiving a file uploaded by a user through a sharing node, the method further comprises:

and judging whether the uploaded file exists in the point-to-point distributed file system or not according to the hash value of the file block of the uploaded file, if not, executing the step of storing the uploaded file in a storage node in the point-to-point distributed file system where the sharing node is located, otherwise, not executing.

Further, still include:

adding the hash value of the file block which is confirmed to be the illegal file in the uploaded files into a blacklist;

when receiving the file that the user uploaded through sharing node, still include:

and judging whether the hash value of the file block of the uploaded file is in the blacklist, if not, executing the step of storing the uploaded file in a storage node in a point-to-point distributed file system where the sharing node is located, otherwise, not executing the step.

Further, when a read request initiated by a reading node to a file stored in the peer-to-peer distributed file system is received, the method further includes:

and judging whether the hash value of the file block of the file requested to be read is in the blacklist, if not, executing the step of initiating a charging request to the reading node according to the cost corresponding to the file requested to be read, otherwise, not executing the step.

In order to achieve the above object, the present invention further provides a content sharing system based on a peer-to-peer distributed file system, including:

the storage module is used for storing the uploaded files in storage nodes in a point-to-point distributed file system where the sharing nodes are located when receiving the files uploaded by the user through the sharing nodes;

the charging module is used for initiating a charging request to the reading node according to the cost corresponding to the file requested to be read when receiving a reading request initiated by the reading node to the file stored in the point-to-point distributed file system;

and the caching module is used for distributing the paid fee to accounts corresponding to the sharing node and the storage node according to a preset rule and caching the file requested to be read to the reading node when the reading node is confirmed to finish fee payment according to the charging request.

Further, the storage module is specifically configured to:

Further, the cache module is specifically configured to:

Further, the storage module is further configured to:

when receiving a file uploaded by a user through a sharing node, judging whether the uploaded file exists in a point-to-point distributed file system or not according to the hash value of a file block of the uploaded file, if not, executing a step of storing the uploaded file in a storage node in the point-to-point distributed file system where the sharing node is located, otherwise, not executing.

Drawings

FIG. 1 is a flowchart illustrating a content sharing method based on a peer-to-peer distributed file system according to an embodiment of the present invention;

fig. 2 is an architecture diagram of an authentication transaction system.

Detailed Description

The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.

The present invention relates to three types of users: sharer, reader and depositor. The sharer can store the content to the system without limit, the storer is responsible for providing and maintaining the storage resource, the divided file blocks of the system are stored for the reader to read, and the reader needs to obtain the resource shared/stored by other people in a payment mode. Each user must provide his own unique identification id, which may be a passport number or an identification number, to ensure a real-name system. For sharers and depositors, the system may require the user to provide a corresponding bank account to receive economic benefits. For the reader, the system does not mandate that a bank account be provisioned if it is not determined that a payment is to be made to subscribe to a resource.

In particular, the sharer is an optional user role in the system to which the present invention relates. A sharee can store content to the system without restriction. For shared content, a sharer has the right to price the shared content (the price can be zero), and after the pricing is checked, other roles can access the shared content only by paying. The system may require the user to provide a corresponding bank account to accept the corresponding economic benefit.

The reader is another optional user role. If a reader wants to access the shared content of a certain sharer, the reader needs to obtain a corresponding hash value by paying after verifying that the resource is available through the server, so as to access the content.

The depositor is responsible for providing and maintaining storage resources and storing the document pieces divided by the system for the reader to read

The reader needs to obtain the resources shared/stored by others by paying

The user can install a file browser program provided by the platform, and after obtaining the user's consent, the user can establish a point-to-point distributed file system node on the host computer and connect with the host network. The user may choose to share any content to the public, such as: images, audio, articles, etc. The content is encrypted, sliced, hashed, and then distributed to other eligible and willing to be stored user machines on the network. If other people want to access the content shared by the user, the user's consent needs to be obtained through the system platform and a certain fee needs to be paid. The specific cost is determined by the participation of the user and the system, and the user in charge of storage, the content sharing user and the platform can be divided according to a certain proportion.

According to the above mode, an embodiment of the present invention provides a content sharing method based on a peer-to-peer distributed file system, as shown in fig. 1, the method includes:

and S1, when receiving the file uploaded by the user through the sharing node, storing the uploaded file in a storage node in the point-to-point distributed file system where the sharing node is located.

S2, when a reading request initiated by a reading node to the file stored in the point-to-point distributed file system is received, initiating a charging request to the reading node according to the cost corresponding to the file requested to be read;

and S3, when the reading node is confirmed to complete the payment according to the charging request, distributing the paid fee to accounts corresponding to the sharing node and the storage node according to a preset rule, and caching the file requested to be read to the reading node.

Accordingly, an embodiment of the present invention provides a content sharing system based on a peer-to-peer distributed file system, including:

The invention can be realized based on a system platform consisting of a point-to-point distributed file system, an authentication transaction system, a content market and a file browser, and the system platform is respectively specifically introduced as follows:

(1) point-to-point distributed file system

The invention adopts a point-to-point distributed File System which mainly comprises an InterPlanetary File System (IPFS), is a global storage network based on going to the center, performs dispersed storage, and aims to utilize idle resources to store and access files, websites, data and other applications. The File System implementation includes a p2p (peer to peer) peer-to-peer network, a bit stream distribution protocol, an SFS-based data transfer protocol (Self-Certifying File System, SFS for short), and the like.

Unlike the HyperText Transfer Protocol (HTTP), this type of peer-to-peer distributed file system is not based on domain name addressing, but content addressing, and it calculates a hash value from the content of a resource, and this value directly reflects the content of this resource, and for a file object, it first divides into a plurality of data blocks, and then calculates the hash value of each data block, and the hash values of these data blocks represent the hash value of this file. The system provides a Distributed Hash Table (DHT) similar to a hash table through a p2p peer-to-peer network. The mapping relation information of all data blocks and the target node is stored in the DHT and maintained by all nodes in the network. Any participating node can effectively retrieve the value corresponding to a given key, and each node is responsible for a small part of routing and data storage. Taking IPFS as an example, a file object is first divided into a plurality of data blocks according to the initialization definition of the IPFS protocol, the current initialization definition of IPFS is 256kb at the maximum, and a 1025k file a is divided into 5 data blocks according to the definition, wherein the first 4 data blocks are 256kb each, and the last 1 data block is 1 kb. The hash value of each chunk is then calculated, and the hash values of the 5 chunks represent the hash value of the file.

After the sharer publishes the file, the file browser serves as a node to help the user divide the file into a plurality of file blocks, and the hash value of the file is uploaded to the authentication transaction system. When a reader wants to obtain files for payment, the nodes quickly find the corresponding nodes through the DHT by the algorithm to obtain the data of the file blocks, and meanwhile, the file blocks are cached locally. The reader, while obtaining the document, also temporarily becomes a depositor itself.

The file system described in the present invention does not backup files without limitation, and the stored files have two categories, one is important and the other is cache. Locally uploaded files are important and data obtained by other nodes is cached. The difference between the two file categories is that important files are always stored in the nodes, and cache files are life-cycle and are cleaned by the garbage collector. Therefore, the effective utilization of the equipment is ensured to the maximum extent, the storage space is not wasted, and meanwhile, the point-to-point output speed is high because the data is cached on the reading node.

(2) Authentication transaction system

The transaction authentication system is an important component for ensuring that the system can work normally. Unlike decentralized point-to-point distributed file systems, authentication systems are centralized. The authentication transaction system is used as an upper layer application and mainly used for providing services such as user registration verification service, user transaction data retention, payment gateway and the like. As shown in fig. 2, the main processing scenarios of the service layer are as follows:

generating node ID

Any newly created child node wants to join the network and must hold a node ID, otherwise the master network rejects the connection.

After the registration is completed, the new user receives the generated node ID generated by the Business (Business) layer, and the user information is stored by the Persistence (Persistence) layer. Only through the node ID, the new user can join the main network to start browsing, publishing or storing information.

Globally, all files that exist in the system can be considered to be in a namespace. To ensure security, the system is SFS-based requiring that all nodes must generate a pair of public and private keys. The file decryption can be carried out only by simultaneously possessing the public key and the private key. The system designed by the invention adopts SFS to embed private key information into the file name, namely self-verifying the file name. The key management is not required to be realized in the file system, the key management function is added into the rule for naming the files by the nodes, and the nodes can select the encryption mode according to the needs.

Examples are:

1. the node generates its own public key and private key pair and carries out Hash operation on the public key, namely node ID

2. Each user is assigned a variable namespace, e.g./sfs

3. A user can issue an object signed by own private key in the path

Such as: /sfs/XLF2apQ4jD5Udex3xp1KBgeSRhemUtaA8Gv

4. When other users acquire the object, the other users can detect whether the signature is matched with the node ID and the public key, verify the authenticity of the object issued by the user and achieve the acquisition of a variable state.

Maintenance of platform order

The system related to the invention has requirements on the shared content of the user, namely the shared content is required to be original legal content. The system will have a record of the hash value of the content uploaded by each user. Based on the content addressing technology, as long as the files are the same, the hash values of the files are the same, and the system maps and stores the user node ID and the file hash based on the hash values, so that the platform order is maintained.

For example, one:

if someone attempts to upload original results of plagiarism others, before uploading, the file browser transmits the file hash to the authentication transaction system for matching. If the authentication system finds that the same file hash exists under other user names, the authentication system refuses to issue again.

Example two:

if some finds that other users maliciously reveal the original achievements of the users and issue the original achievements in advance, the original achievements can be reported through the browser. After the evidence with legal effectiveness is submitted, the system can remove the binding between the malicious user and the file and restore the rights and interests of the original personnel.

Example three:

if someone pays for some original result, the private key of the file is handed to others without authorization, and when others try to access the file through a file browser, the authentication transaction system can detect the mapping relation stored in the persistent layer and judge whether the login user has the browsing authority, and if not, the access is refused.

Payment gateway

For the purpose of business model, the reader needs to pay a certain amount, and the authenticated transaction system integrates payment gateways (Paibao, WeChat Payment, Unionpay, Visa, MasterCard, JCB, etc.) of all the large transaction systems, and after the payment is collected, the payment is distributed to sharers and depositors according to rules.

A file browser is a user-oriented terminal. The browser provides interfaces for registering, logging in, managing personal information, configuring nodes, uploading/downloading files, accessing content markets, etc. Users need to read and accept regulatory compliance licensing agreements when opening software and registering. If any illegal content is found through auditing, the platform has the right to shield and delete the corresponding content through the App client, so that the aim of shielding illegal contents from being spread by lawbreakers is fulfilled.

The software can help a user to build a node of the point-to-point distributed browser in a one-click mode, and the user is helped to configure the node through an interactive UI, wherein the configuration comprises a storage resource giving-out time limit, a giving-out storage resource size and the like.

Whenever the user connects to the network, it connects to the authenticated transaction system and synchronizes the information. When someone reports that a file is problematic in the network, the hash value of each file block of the reported file with the problem is recorded by the verification transaction system and added into a blacklist, and the file browser synchronizes the information to the local part during synchronization. And if the hash value of the file to be shared of the user is matched with the hash value in the blacklist, the system refuses the file uploading.

If the file is uploaded, when a reader requests to access the corresponding file, the system can also match through a blacklist, and if the file belongs to an illegal file, the file is refused to be presented to a user.

Meanwhile, the client deletes the illegal contents determined according to the country and the region, the enterprise should keep a backup file in the business level, and the user can claim the illegal contents by carrying the identity of the user or the valid certificate of the enterprise if the user has an objection.

(3) Content marketplace

The method and the system can be used for constructing a content market, wherein the content market is a free platform for publicizing content for sharers, which is provided by a system platform, and the file browser provides a browsing interface of the market. The user can search for favorite contents through the market and then pay for purchase. On the basis of obtaining the consent of the user, the system collects the search history of the user and provides content recommendation for the user through deep learning. It is to be understood that the platform is a content-based e-commerce mall. Since the system related to the present invention is not innovative in this respect, there are many specific implementations, which are not described herein.

(4) File browser

If the file is uploaded, when a reader requests to access the corresponding file, the system can also match through a blacklist, and if the file belongs to an illegal or illegal file, the file is refused to be presented to the user.

For newly uploaded files, the file browser does not immediately propagate the newly uploaded files to the network, but needs to pass through the content auditing system before the newly uploaded files are formally released. The content auditing system is used for protecting copyright owners based on image and character comparison, and the specific implementation scheme of the system is as follows:

when a user uploads an attempt to upload a file, the file browser searches whether the same content already exists through a file hash, and if so, the file browser directly rejects the file. If the same file hash does not exist, specific contents are analyzed and compared for similarity.

The content auditing system can be used for video, image, audio and text files in different modes. The following are described respectively:

images

And extracting the finger print of the image through a perceptual hashing algorithm.

The hash value of each uploaded picture is stored in the database, and when a new picture is uploaded, the system calculates the Hamming distance between the two pictures. Generally, hamming distances less than 10 are considered similar, whereas differences are considered greater.

Audio frequency

And extracting finger ping information of the image by a Shazam-like algorithm.

The similarity of the two audios is judged by comparing the distance of the finger print.

Video

The video can be regarded as a combination of image and audio essentially, the system can split the video file, and the video itself is detected by using the image and audio similarity detection technology, which is not described herein again.

Text

The text similarity detection adopts a simhash algorithm

TF-IDF: TF (word frequency), IDF (inverse word frequency)

And judging the similarity of the texts by comparing the Hamming distances.

After the content verification is completed, the client deletes the illegal content which is determined to be illegal aiming at the country and the region, the enterprise should keep a backup file at the commercial level, and the user can claim the illegal content by carrying the identity of the user or the valid certificate of the enterprise if the user has an anomaly.

The charging mode of the platform constructed based on the embodiment is as follows:

the principle of the charging mode is that the reader pays X, the sharer profit N, the platform profit P and the depositor profit Q, wherein X is N + P + Q. However, based on different situations, the platform can be adjusted according to specific situations, and other types of payment modes such as membership and the like are not excluded.

Examples are:

xiaoming shares a movie authored by oneself, and is priced into 10 yuan. The platform specifies a video class author profit rate of 70% and a commission of 10%.

When the movie video file is obtained by paying a small payment of 10 yuan, the small payment is regarded as the sharer income 10 × 90% by 70%, namely 6.3 yuan. The video file is divided into 10 file blocks in total. When the Xiaoqiang obtains the file, 5 file blocks are obtained from a Xiaoming computer, and 1 file block is obtained from other users A, B, C, D and E respectively. Xiaoming as stock, earnings 10 × 90% × 30% × 5/10 ═ 1.35 yuan, a, B, C, D, E each gave 0.27 yuan.

The total profit scenario is as follows:

platform revenue: 1 Yuan

Little obvious income: 6.3+1.35 ═ 7.65 yuan

A, B, C, D, E: each 0.27 membered

1+7.65+0.27 x 5 ═ 10 dollars, i.e., 10 dollars for small strong payments.

The above example is only one of a specific implementation.

The invention has the following advantages:

1. integrating and utilizing idle memory resources

The business model designed by the invention is essentially a shared economy, namely, users are encouraged to go out of idle disks to provide storage resources for users in need, and content distribution is carried out through a platform, so that the two parties can benefit simultaneously. The platform can release the original content of the user who has the original resource release and price the user, and can contribute a part of the hard disk of the user who has the idle storage resource, and the platform can return the received cost to the author and the person who provides the storage resource according to a certain proportion when the user who has the demand pays after seeing the content that the user wants on the platform.

2. Knowledge copyright protection

The platform users related to the invention are required to be in a real name system, and the users are required to perform through appointing App when uploading and downloading files. The system refuses to post the same content that has already been posted a second time. While the user needs to read and accept regulatory compliance license agreements at the time of registration. If any illegal content is found through auditing, the platform has the right to delete the corresponding content through the App, so that the aims of shielding illegal contents from being spread by lawbreakers and protecting the intellectual property of original authors are fulfilled.

3. Financial risk

The current mainstream practice is to incentivize users to fulfill contracts through ICOs, but it can bring uncertain financial risks to society and countries. The invention adopts a sharing economy mode without ICO.

4. High speed

The system related to the invention is faster than the general traditional website, because the transmission stage is based on the point-to-point transmission technology of the p2p network.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A content sharing method based on a point-to-point distributed file system is characterized by comprising the following steps:

2. The content sharing method based on the peer-to-peer distributed file system according to claim 1, wherein the storing the uploaded file in a storage node in the peer-to-peer distributed file system where the sharing node is located specifically comprises:

3. The content sharing method based on the peer-to-peer distributed file system according to claim 2, wherein caching the file requested to be read to the reading node specifically includes:

4. The content sharing method based on the peer-to-peer distributed file system according to claim 2, wherein when receiving the file uploaded by the user through the sharing node, the method further comprises:

5. The method for content sharing based on peer-to-peer distributed file system according to any of claims 2 to 4, further comprising:

6. The peer-to-peer distributed file system based content sharing method according to claim 5, wherein when receiving a read request initiated by a reading node for a file stored in the peer-to-peer distributed file system, the method further comprises:

7. A content sharing system based on a peer-to-peer distributed file system, comprising:

8. The peer-to-peer distributed file system based content sharing system as claimed in claim 7, wherein said storage module is specifically configured to:

9. The content sharing system according to claim 8, wherein the cache module is specifically configured to:

10. The peer-to-peer distributed file system based content sharing system as claimed in claim 8 or 9, wherein the storage module is further configured to: