CN112650721A - File storage method, device, system and equipment - Google Patents
File storage method, device, system and equipment Download PDFInfo
- Publication number
- CN112650721A CN112650721A CN202011596322.9A CN202011596322A CN112650721A CN 112650721 A CN112650721 A CN 112650721A CN 202011596322 A CN202011596322 A CN 202011596322A CN 112650721 A CN112650721 A CN 112650721A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- server
- stored
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 230000005540 biological transmission Effects 0.000 claims description 34
- 238000012360 testing method Methods 0.000 claims description 17
- 239000012634 fragment Substances 0.000 claims description 14
- 230000000977 initiatory effect Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 description 22
- 238000004590 computer program Methods 0.000 description 11
- 238000013467 fragmentation Methods 0.000 description 11
- 238000006062 fragmentation reaction Methods 0.000 description 11
- 230000007246 mechanism Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000003999 initiator Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The application provides a file storage method, device, system and equipment, and by building a file storage system with a multi-directory server architecture, the file storage system is guaranteed to realize efficient index service, so that more efficient file storage service is provided. The method is applied to a file storage system and comprises the following steps: determining a file to be stored, wherein the file to be stored is a file which is requested by a client and stored in a file storage system; distributing corresponding target directory servers for files to be stored in a directory server set, wherein the directory server set comprises different directory servers preset by a file storage system; the method comprises the steps of sending an acquired file to be stored to a target storage server for storage through a target directory server, and establishing index data between the file to be stored and the target storage server, wherein the target storage server is distributed from a storage server set by the target server, and the storage server set comprises different storage servers.
Description
Technical Field
The present application relates to the field of storage, and in particular, to a file storage method, apparatus, system, and device.
Background
In order to facilitate storage management of files, a file storage system is generally configured for use by users such as consumers or personnel in a company, and the users can upload files to a server of the file storage system for storage at a client side.
On the server side, in order to facilitate the hierarchical processing of the work, the file storage system specifically comprises a directory server and a storage server, the directory server is used for centrally processing the storage request initiated by the client, the storage server capable of storing the current file is allocated, the storage server is used for storing the file uploaded by the client, and the file query and download links are similar to the links. Thus, the file storage service of the file storage system is realized based on the index service provided by the directory server.
In the research process of the related art, the inventor finds that with the file storage service provided by the file storage system, as the occurrence frequency of large files is gradually increased nowadays, the directory server is becoming overwhelmed, and the situation of low processing efficiency often occurs.
Disclosure of Invention
The application provides a file storage method, device, system and equipment, and the file storage system with a multi-directory server architecture is built, so that the file storage system can realize efficient index service, the condition that the processing efficiency of a directory server is low in the prior art is avoided, and more efficient file storage service is provided.
In a first aspect, the present application provides a file storage method, including:
determining a file to be stored, wherein the file to be stored is a file which is requested by a client and stored in a file storage system;
distributing corresponding target directory servers for files to be stored in a directory server set, wherein the directory server set comprises different directory servers;
the method comprises the steps of sending an acquired file to be stored to a target storage server for storage through a target directory server, and establishing index data between the file to be stored and the target storage server, wherein the target storage server is distributed from a storage server set by the target server, and the storage server set comprises different storage servers.
With reference to the first aspect of the present application, in a first possible implementation manner of the first aspect of the present application, allocating, in a directory server set, a corresponding target directory server for a file to be stored includes:
evaluating the network transmission quality between different directory servers and the client respectively through the transmission test of the test data packet;
and selecting the directory server corresponding to the best network transmission quality as the target directory server.
With reference to the first aspect of the present application, in a second possible implementation manner of the first aspect of the present application, the target storage server includes a plurality of sub-target storage servers, and sends the obtained file to be stored to the target storage server for storage through the target directory server, and establishes index data between the file to be stored and the target storage server, including:
the method comprises the steps that an obtained file to be stored is fragmented through a target directory server to obtain a plurality of fragmented files;
respectively sending the plurality of fragmented files to a plurality of subdirectory storage servers for distributed storage;
index data between a plurality of fragmented files and a plurality of sub-target storage servers is established.
With reference to the first aspect of the present application, in a third possible implementation manner of the first aspect of the present application, the method further includes:
and synchronizing the index data between the file to be stored and the target storage server to other directory servers in the directory server set through the target directory server.
With reference to the first aspect of the present application, in a fourth possible implementation manner of the first aspect of the present application, sending, by a target directory server, an obtained file to be stored to a target storage server for storage includes:
receiving a file to be stored uploaded by a client by a directory server;
and sending the file to be stored to a target storage server for storage through the target server.
With reference to the first aspect of the present application, in a fifth possible implementation manner of the first aspect of the present application, the file storage system is built based on a blockchain platform, and different directory servers and different storage servers are all different blockchain link point devices in the blockchain platform.
With reference to the first aspect of the present application, in a sixth possible implementation manner of the first aspect of the present application, the method further includes:
receiving a downloading request, wherein the downloading request is used for requesting to download a file to be stored from a file storage system;
determining a target storage server for storing the file to be stored according to the index data;
and providing a downloading service of the file to be stored to the client initiating the downloading request through the target storage server.
In a second aspect, the present application provides a file storage apparatus, the method comprising:
the device comprises a determining unit, a storage unit and a processing unit, wherein the determining unit is used for determining a file to be stored, and the file to be stored is a file which is requested by a client and stored in a file storage system;
the device comprises a distribution unit, a storage unit and a processing unit, wherein the distribution unit is used for distributing a corresponding target directory server for a file to be stored in a directory server set, and the directory server set comprises different directory servers;
the sending unit is used for sending the obtained file to be stored to a target storage server for storage through a target directory server, wherein the target storage server is obtained by the target server in a distribution mode from a storage server set, and the storage server set comprises different storage servers;
and the establishing unit is used for establishing index data between the file to be stored and the target storage server.
With reference to the second aspect of the present application, in a first possible implementation manner of the second aspect of the present application, the allocating unit is specifically configured to:
evaluating the network transmission quality between different directory servers and the client respectively through the transmission test of the test data packet;
and selecting the directory server corresponding to the best network transmission quality as the target directory server.
With reference to the second aspect of the present application, in a second possible implementation manner of the second aspect of the present application, the target storage server includes a plurality of sub-target storage servers, and the sending unit is specifically configured to:
the method comprises the steps that an obtained file to be stored is fragmented through a target directory server to obtain a plurality of fragmented files;
respectively sending the plurality of fragmented files to a plurality of subdirectory storage servers for distributed storage;
the establishing unit is specifically configured to:
index data between a plurality of fragmented files and a plurality of sub-target storage servers is established.
With reference to the second aspect of the present application, in a third possible implementation manner of the second aspect of the present application, the apparatus further includes a synchronization unit, configured to:
and synchronizing the index data between the file to be stored and the target storage server to other directory servers in the directory server set through the target directory server.
With reference to the second aspect of the present application, in a fourth possible implementation manner of the second aspect of the present application, the sending unit is specifically configured to:
receiving a file to be stored uploaded by a client by a directory server;
and sending the file to be stored to a target storage server for storage through the target server.
With reference to the second aspect of the present application, in a fifth possible implementation manner of the second aspect of the present application, the file storage system is built based on a blockchain platform, and different directory servers and different storage servers are all different blockchain link point devices in the blockchain platform.
With reference to the second aspect of the present application, in a sixth possible implementation manner of the second aspect of the present application, the apparatus further includes a downloading unit, configured to:
receiving a downloading request, wherein the downloading request is used for requesting to download a file to be stored from a file storage system;
determining a target storage server for storing the file to be stored according to the index data;
and providing a downloading service of the file to be stored to the client initiating the downloading request through the target storage server.
In a third aspect, the present application further provides a file storage system, where the system includes a directory server set and a storage server set, the target server set includes different directory servers, the storage server set includes different storage servers, and the system is configured to execute the method provided by the first aspect of the present application or any possible implementation manner of the first aspect of the present application.
In a fourth aspect, the present application further provides a file storage device, which includes a processor and a memory, where the memory stores a computer program, and the processor executes the method provided by the first aspect of the present application or any one of the possible implementation manners of the first aspect of the present application when calling the computer program in the memory.
In a fifth aspect, the present application further provides a computer-readable storage medium storing a plurality of instructions, which are suitable for being loaded by a processor to perform the method provided in the first aspect of the present application or any one of the possible implementation manners of the first aspect of the present application.
From the above, the present application has the following advantageous effects:
aiming at a file storage scene, in the file storage system built by the application, a multi-directory server architecture is deployed, after a client side is determined to request a file to be stored in the file storage system, a corresponding target directory server is allocated for the file to be stored in a directory server set, the obtained file to be stored is sent to the target storage server through the target directory server to be stored, index data between the file to be stored and the target storage server is established, wherein the target storage server is allocated from the storage server set by the target server, and in the file storage process, as the selected target directory server is a directory server suitable for processing the index service of the current file to be stored, the file storage system can realize the index service, so that the situation that the directory server in the prior art has low processing efficiency is avoided, more efficient file storage service is provided, and better user experience is achieved.
Drawings
FIG. 1 is a schematic diagram of an architecture of a file storage system of the present application;
FIG. 2 is a schematic flow chart of a file storage method according to the present application;
FIG. 3 is a schematic flow chart illustrating the allocation of a target directory server according to the present application;
FIG. 4 is a schematic flow chart illustrating processing of a file to be stored according to the present application;
FIG. 5 is a schematic diagram of a structure of a storage device according to the present application;
FIG. 6 is a schematic structural diagram of a storage device according to the present application.
Detailed Description
First, before the present application is introduced, the relevant contents of the present application with respect to the application background will be described.
The file storage method, the file storage device and the computer readable storage medium can be applied to file storage equipment, and the file storage system with a multi-directory server architecture is built, so that the file storage system can realize efficient index service, and more efficient file storage service is provided.
In the file storage method provided by the application, an execution main body may be a file storage apparatus, or different types of file storage apparatuses such as a server device, a physical host, or a User Equipment (UE) integrated with the apparatus, where the apparatus may be implemented in a hardware or software manner, the UE may specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a palmtop computer, a desktop computer, or a Personal Digital Assistant (PDA), and the device may also be configured in a device cluster form.
The file storage device is not particularly a storage server which is allocated by a file storage system for a current file to be stored and is used for storing the file to be stored, but refers to a device which applies the file storage method of the present application.
Referring to a schematic structural diagram of the file storage system shown in fig. 1, in the present application, the built file storage system is differentiated according to functions and includes two parts, namely an index service and a storage service, and corresponding to the index service, compared with a single directory server architecture in the existing file storage system, the file storage system introduces a multi-directory server architecture, so that a target service server suitable for processing can be allocated to a current file to be stored according to actual conditions, and a more efficient load balancing mechanism is realized.
As shown in fig. 1, in the file storage system, a plurality of directory servers and a plurality of storage servers may form a corresponding server set, and for storage server sets corresponding to different target servers, there may be a correspondence relationship that is independent from, overlapped with, and contained in, or integrated with each other (for example, in fig. 1, the storage server set a and the storage server set B, and the storage service set C and the storage server set D are in two overlapping states).
The number of directory servers and storage servers shown in fig. 1 is merely an example, and should not be limited.
Next, a file storage method provided by the present application will be described.
Fig. 2 shows a schematic flow chart of the file storage method of the present application, and as shown in fig. 2, the file storage method provided by the present application may specifically include the following steps:
step S201, determining a file to be stored, wherein the file to be stored is a file which is requested by a client to be stored in a file storage system;
step S202, distributing corresponding target directory servers for files to be stored in a directory server set, wherein the directory server set comprises different directory servers;
step S203, sending the acquired file to be stored to a target storage server for storage through a target directory server, and establishing index data between the file to be stored and the target storage server, wherein the target storage server is distributed from a storage server set by the target server, and the storage server set comprises different storage servers.
As can be seen from the embodiment shown in fig. 2, in a file storage system constructed in the present application, for a file storage scenario, a multi-directory server architecture is deployed, and after a client requests a file to be stored in the file storage system, a corresponding target directory server is allocated to the file to be stored in a directory server set, the obtained file to be stored is sent to the target storage server by the target directory server for storage, and index data between the file to be stored and the target storage server is established, where the target storage server is allocated from the storage server set by the target server, and in a storage flow of the file, because the selected target directory server is a directory server suitable for processing an index service of the current file to be stored, it is ensured that the file storage system can implement an efficient index service, therefore, the situation that the processing efficiency of the directory server is low in the prior art is avoided, more efficient file storage service is provided, and better user experience is achieved.
The following detailed description of the steps shown in fig. 2 and the specific embodiments that may be adopted in practical applications is continued with reference to the architecture of the file storage system shown in fig. 1:
it can be understood that, in the present application, the file storage method provided is described in terms of a file storage system, and in practical applications, the file storage method may be executed by one node in the file storage system, and more likely, executed by different nodes in the file storage system in a cooperative manner, and signaling interaction between different nodes is also involved to implement cooperation between nodes, and the content of the signaling interaction is not repeated except for the content of steps involved in the file storage method of the present application.
In practical application, a file to be stored, which is requested to be stored in a file storage system by a client, is determined, and the method mainly corresponds to two application scenes.
One is that the client has reported a storage request, and carries the file to be stored in the request information of the storage request, for example, in some network disk services, the user can log in a user account and directly upload and store the file, and at this time, the file to be stored can be stored in a node, such as a directory server or other nodes, in the file storage system that receives the storage request.
And the other method is that the client firstly reports a storage request, after the file storage system allocates the target directory server and the target storage server, the client is instructed to upload the file to be stored to the target storage server under the scheduling of the directory server, or the target directory server receives the file to be stored uploaded by the client and forwards the file to be stored to the target storage server for storage.
Thus, it can be seen that, in step S201, determining a file to be stored, which may refer to determining a file to be stored for which a client is to initiate a storage operation, and receiving and storing the file to be stored after the file storage system determines a storage policy of the file to be stored; or the client side may also refer to the file to be stored which is uploaded to the file storage system, and the file to be stored which is already in the system is continuously deployed with a subsequent storage policy.
The file to be stored may be a file of different file types such as a document, a table, a picture, a video, an audio, or a compressed packet, and the file type of the file may be specifically adjusted according to actual needs, which is not limited herein.
After determining the current file to be stored, the file storage system may allocate an adapted target directory server or even a target storage server to the file storage system, so as to implement an adapted high-quality index service.
In the multi-directory server architecture of the present application, the allocation policy of the target directory server may be specifically set according to different adaptation indexes.
For example, if the file storage system faces a user in a larger area range, the file storage system may configure directory servers in different places, and after determining a file to be stored, may select a directory server closest to a client initiating the file to be stored as a target directory server, or select a directory server closest to a node storing the file to be stored in the current system as a target directory server;
for another example, if the file storage system considers the influence of the computing resources and the storage resources on the directory index service, the current computing resources and/or the current storage resources of each directory server in the target server set can be computed, and the directory server with the highest free resource amount or free resource proportion is used as the target directory server;
for another example, if the file storage system considers the transmission speed, a directory server with the highest transmission speed with the client initiating the file to be stored may be selected from the directory server set as the target directory server, or a directory server with the highest transmission speed with a node storing the file to be stored in the current system may be selected as the target directory server;
taking a storage scenario in which the client does not upload the file to be stored yet, referring to fig. 3, which is a schematic flow diagram of allocating the target directory server in the present application, the allocation of the target directory server may include the following steps S3101 and S302:
step S301, evaluating the network transmission quality between different directory servers and clients respectively through the transmission test of test data packets;
corresponding to the distribution factor with the transmission speed as the target directory server, a test data packet may be configured, where the data packet is used to test the transmission speed between the client and the directory server, and the data packet may be a data packet configured for the test, or may also be a data packet that may be transmitted in other services between the client and the directory server.
The file storage system can trigger the client to send the test data packet to each directory server of the system, or the client side relates to an application program of the file storage system, and can also trigger the sending of the test data packet to each directory server of the system by itself.
In each test data packet transmission, the time length used in the transmission process, namely the transmission time length, can be determined by acquiring the transmission time point of the client side and the acquisition time point of the directory server.
Secondly, other influencing factors such as network fluctuation, packet loss and the like may also occur in the transmission process, so that the network transmission quality between the current client and different directory servers can be evaluated correspondingly to the reference factors influencing the network transmission quality by combining the evaluation values and weights of different reference factors, and the evaluation values and weights can be used as quantitative information of the transmission speed.
Step S302, select the directory server corresponding to the best network transmission quality as the target directory server.
After the network transmission quality between the client and different directory servers is evaluated, the directory server corresponding to the best network transmission quality can be used as the target directory server, so that efficient index service is achieved, and efficient file storage service is achieved.
Of course, in practical applications, the allocation policy of the target directory server may not only adopt the above mentioned allocation policy, but also configure other allocation policies according to practical conditions, and the specific implementation is not limited herein.
And after the target directory server is allocated, the acquired file to be stored can be sent to the target storage server for storage through the target directory server.
For the directory server, a storage server range (storage server set) capable of performing indexing is preset, and after the file to be stored which needs to be processed by the directory server is determined, the target directory server can allocate a target storage server for the file to be stored from the storage server set, and the target storage server stores the file to be stored.
In the allocation process of the target storage server, it can be understood that, generally, the storage server having the most free storage resources or the highest proportion of free storage resources is preferred as the allocation result, and of course, other allocation policies are not excluded, and the specific details are not limited herein.
In addition, it can be understood that, in practical application, there may be two or more target storage servers, so as to achieve a situation that a plurality of storage servers store files to be stored, achieve an effect of improving the disaster tolerance capability of the system, and avoid the influence of a single storage server fault on the file storage service of the system.
Meanwhile, in order to further improve the response efficiency of the storage service, a fragmentation mechanism of the file to be stored is also introduced in the application.
Referring to a schematic flow chart of the present application for processing a to-be-stored file shown in fig. 4, in the present application, the processing of the to-be-stored file implemented by the target directory server may specifically include the following steps S401 to S403:
step S401, fragmenting the acquired file to be stored through a target directory server to obtain a plurality of fragmented files;
it can be understood that after the target directory server obtains the file to be stored, for example, the target directory server may receive the file to be stored uploaded by the client, or the file to be stored is called from a node in the system where the file to be stored is stored, and the file to be stored may be fragmented; or the client performs fragmentation processing locally and then uploads.
Fragmentation can be understood as splitting, a file to be stored is decomposed into a plurality of subfiles called fragmented files, and after the fragmented files are combined, the original file to be stored can be obtained.
It should be understood that these fragmented files, the file to be stored, are fragmented under different fragmentation scopes, and these fragmentation scopes may be independent from each other, and there may be overlapping situations.
The fragmentation range can be configured in a random, preset or other manner, and if an overlapping mechanism of the fragmentation range is introduced, it is worth noting that content loss caused by abnormal situations such as file loss can be further reduced due to the fact that the content of the files is repeated among the fragmentation files. If the A fragment file is lost, because the B fragment file has the X file content of the A fragment file, the X file content obviously has the existence of the B fragment file, and the content loss is not caused.
Step S402, respectively sending the plurality of fragmented files to a plurality of subdirectory storage servers for distributed storage;
similarly, the file storage system of the present application may also disperse a target storage server storing a file to be stored into a plurality of storage servers (sub storage servers), where each sub storage server may store one or more fragmented files.
In this case, the target directory server may then initiate distributed storage based on its communication connection with a plurality of allocated sub-target storage services, and distribute the fragmented files to these sub-target storage servers.
Through the distributed storage mechanism, the transmission speed of the file to be stored to the target storage server can be increased, the data security of the file to be stored in the file storage server can be further improved, and the data loss caused by the abnormity of the individual storage server can be reduced.
Step S403, establishing index data between the plurality of fragmented files and the plurality of sub-target storage servers.
The directory server, as can be understood, the index service provided by the directory server can be implemented by index data, in which the corresponding relationships between different files to be stored and different storage servers are recorded, so that after the target storage service of the current file to be stored is determined, the storage relationship can be embodied in the index data, and if the query or download of the file to be stored occurs subsequently, the query or download requirement can be responded according to the index data.
Specifically, the index data may be configured in a data table manner, or may also be configured in another data form, which is not limited herein.
Meanwhile, when the fragmentation mechanism and the multi-target storage server mechanism are applied, the storage relationship can be stored in the index data, for example, the storage relationship is updated into the total index data, or an index data is newly created, so as to establish the current index data.
Further, in consideration of subsequent query and download scenarios, in some application scenarios, for a certain file to be stored, the network transmission quality between the original target directory server and the client initiating the query and download may change along with the influence of actual conditions, for example, the network environment connected to the client is updated, the location of the client is more, and the like.
Alternatively, the directory server that originally processed the file to be stored may not be considered the directory server that received the query, download request.
At this moment, in order to save the cost required by the current directory server to call the related index data from the directory server which originally processes the file to be stored, the index data indicating the storage relationship between the file to be stored and the target storage server can be synchronized to other directory servers in the directory server set in advance, so that the subsequent index data can be directly called, the high processing efficiency of the index service is guaranteed, meanwhile, under the synchronization mechanism, the data security of the index data can be further improved, the condition that a single directory server is abnormal or the data is lost is avoided, and the disaster tolerance capability of the file storage system is improved.
Subsequently, in order to further improve data security, a block chain technology can be introduced, or a file storage system is built on the basis of a block chain platform, and for this reason, a directory server and a storage server are both block chain link point devices in the block chain platform, and at this time, a file to be stored is locally used as data to be linked up for the link up, so that the purpose of storing the file to be stored in the file storage system is achieved.
In addition, for the called client, it may also be a device of the blockchain link point in the blockchain platform even included in the file storage system, for example, the client may perform the fragmentation processing of the file to be stored, and of course, it may also be a device outside the blockchain platform, which is not described herein again specifically.
In this application scenario, it is easily understood that, based on the blockchain technology, when data related to the above-mentioned content, such as the file to be stored, the index data, and the like, is transmitted in the blockchain platform, processing for calculating a Hash value based on a Hash (Hash) algorithm may be involved, when an initiator of the data sends the data, the Hash value of the data may be calculated through the Hash algorithm, and the Hash value may be uplinked (for example, uplinked to a directory server or a storage server), when a receiver of the data receives the data, the Hash value of the data may be calculated through the Hash algorithm, since the Hash algorithm is an algorithm that can only encrypt and cannot decrypt, the Hash value calculated by a certain data through the Hash algorithm may be considered to be unique, at this time, if the receiver of the data finds that the Hash value calculated by itself is consistent with the Hash value on the chain through comparison, the data can be regarded as authentic data without being tampered during transmission and with the authenticity thereof.
For example, the hash value is calculated by converting the corresponding data into binary data, and then converting the binary data into a 64-length hexadecimal string using SHA-256 function.
Certainly, besides the verification mechanism implemented by the hash algorithm, the blockchain technology also has other data processing mechanisms for ensuring data security, which are not described herein again specifically, and because the blockchain technology has the characteristics of transparent rules, data security and the like, the file storage system provided by the application is built based on the blockchain platform, which obviously has higher practical value, not only because the blockchain working mechanism can prevent data from being tampered in the data transmission process, and ensures data security, but also because the system architecture and the working mode disclosed by the application in the above contents have higher processing efficiency and system disaster tolerance capability.
After the storage of the file to be stored is completed, the file storage system can also provide query and download services for the user, wherein it is easy to understand that in practical application, the download service may include a query service.
Taking a file downloading service provided by a file storage system as an example, the downloading process generally comprises:
acquiring a downloading request, wherein the downloading request is used for requesting to download a file to be stored from a file storage system;
determining a target storage server for storing the file to be stored according to the index data;
and providing a downloading service of the file to be stored to the client initiating the downloading request through the target storage server.
Specifically, the file storage system may configure a node open to a user, so that the user initiates a download request, such as a directory server or other nodes, and when the download request is obtained, a target directory server that originally processes a file to be stored may be determined by the originally configured index data, and then the target directory server responds to the download request.
Of course, since the client may have previously communicated with the target directory server that originally processed the file to be stored, the client may also directly initiate a download request to the target directory server according to the history (e.g., the history communication address, the history server identifier, etc.).
In the process of responding to the downloading request, the target directory server can determine a corresponding target storage server according to the preset index data, and then the target storage server provides the downloading service for the client.
It should be noted that, this downloading scenario is only an example, and in practical applications, as mentioned above, the index data may be synchronized between directory servers, so that a downloading request may be sent to any directory service storing the index data for response; or, the node receiving the download request may query the directory server for the target storage server to respond, or may feed back the index data to the client according to the download request, and the client initiates the download service to the target storage server according to the index data.
The target storage server provides downloading service for the client, which not only can be understood as that the target storage server directly transmits the file to be stored to the client, but also can be understood as that the file to be stored is indirectly transmitted to the client through forwarding of the directory server, for example, the above-mentioned fragment files of the sub-target storage servers are summarized or even combined by the directory server to obtain the file to be stored, and then transmitted to the client, or the sub-target storage servers directly transmit the stored sub-file to be stored to the client, and the original file to be stored is obtained by combining the client.
For convenience of description, a file storage system built based on a block chain platform is taken as an example, and the above-mentioned fragmentation mechanism is combined to provide a processing flow example of a storage service related to a file to be stored.
1. A client requests a file to upload;
2. the testing network selects the directory server belonging to the best uploading node;
3. the directory server divides files uploaded by the client into pieces (original file pieces are obtained) and stores the pieces into the storage servers to form redundant backup of the divided files (one piece can be stored in a plurality of storage servers), and meanwhile, the directory server calculates hash values of the original files and the file pieces and links storage certificates;
4. after each storage server receives the file fragments, calculating the hash value of the fragments, and comparing the hash value with the hash value of the corresponding file fragments on the chain to ensure that the received fragmented files are real and consistent;
5. the directory server establishes an index directory and links a chain certificate to the index directory;
6. the original directory server synchronizes the index directory to other directory servers and even to the storage server, and the other directory servers compare the received hash value of the index directory with the hash value of the index directory storing the certificate on the link, so as to ensure the truth and consistency;
7. a client initiates a downloading request;
8. selecting a directory server, wherein the directory server provides an index directory to a client;
9. and the client compares the obtained hash value of the index directory with the hash value of the index directory storing the certificate on the link, so as to ensure the truth and the consistency.
10. The client side obtains the fragment files from the storage server according to the index directory, hash calculation is carried out on each fragment file at the client side, and the obtained hash value is compared with the hash value on the chain, so that the authenticity and consistency of the fragment files are ensured; if the individual fragmented files are inconsistent, the backup of the same fragmented file on another storage server can be obtained according to the index directory, and the hash value is compared again until the files are truly consistent;
11. and the client combines the original files according to the fragment files, calculates the hash value, and compares the hash value with the hash value of the original file on the link to ensure the authenticity and consistency.
The above is the description of the file storage method of the present application, and in order to better implement the file storage method provided by the present application, the present application also provides a file storage device.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a file storage device according to the present application, in the present application, a file storage device 500 is applied to a file storage system, and the file storage device 500 may specifically include the following structure:
a determining unit 501, configured to determine a file to be stored, where the file to be stored is a file requested by a client to be stored in a file storage system;
an allocating unit 502, configured to allocate a corresponding target directory server for a file to be stored in a directory server set, where the directory server set includes different directory servers;
a sending unit 503, configured to send the obtained file to be stored to a target storage server for storage through a target directory server, where the target storage server is obtained by allocating the target storage server from a storage server set, and the storage server set includes different storage servers;
the establishing unit 504 is configured to establish index data between the file to be stored and the target storage server.
In an exemplary implementation, the allocating unit 502 is specifically configured to:
evaluating the network transmission quality between different directory servers and the client respectively through the transmission test of the test data packet;
and selecting the directory server corresponding to the best network transmission quality as the target directory server.
In another exemplary implementation manner, the target storage server includes a plurality of sub-target storage servers, and the sending unit 503 is specifically configured to:
the method comprises the steps that an obtained file to be stored is fragmented through a target directory server to obtain a plurality of fragmented files;
respectively sending the plurality of fragmented files to a plurality of subdirectory storage servers for distributed storage;
the establishing unit is specifically configured to:
index data between a plurality of fragmented files and a plurality of sub-target storage servers is established.
In yet another exemplary implementation, the apparatus further includes a synchronization unit 505 for:
and synchronizing the index data between the file to be stored and the target storage server to other directory servers in the directory server set through the target directory server.
In another exemplary implementation manner, the sending unit 503 is specifically configured to:
receiving a file to be stored uploaded by a client by a directory server;
and sending the file to be stored to a target storage server for storage through the target server.
In yet another exemplary implementation, the file storage system is built based on a blockchain platform, and different directory servers and different storage servers are all different blockchain link point devices in the blockchain platform.
In yet another exemplary implementation, the apparatus further includes a downloading unit 506 for:
receiving a downloading request, wherein the downloading request is used for requesting to download a file to be stored from a file storage system;
determining a target storage server for storing the file to be stored according to the index data;
and providing a downloading service of the file to be stored to the client initiating the downloading request through the target storage server.
The present application further provides a file storage device, and referring to fig. 6, fig. 6 shows a schematic structural diagram of the file storage device of the present application, specifically, the file storage device of the present application includes a processor 601, a memory 602, and an input/output device 603, where the processor 601 is configured to implement each step of the file storage method in any embodiment corresponding to fig. 2 to fig. 4 when executing a computer program stored in the memory 602; alternatively, the processor 601 is configured to implement the functions of the units in the embodiment corresponding to fig. 5 when executing the computer program stored in the memory 602, and the memory 602 is configured to store the computer program required by the processor 601 to execute the file storage method in any embodiment corresponding to fig. 2 to 4.
Illustratively, a computer program may be partitioned into one or more modules/units, which are stored in the memory 602 and executed by the processor 601 to accomplish the present application. One or more modules/units may be a series of computer program instruction segments capable of performing certain functions, the instruction segments being used to describe the execution of a computer program in a computer device.
The file storage devices may include, but are not limited to, a processor 601, memory 602, input output devices 603. It will be appreciated by those skilled in the art that the illustration is merely an example of a file storage device and does not constitute a limitation of a file storage device, and that the file storage device may include more or less components than those illustrated, or some components may be combined, or different components, for example, the file storage device may further include a network access device, a bus, etc., and the processor 601, the memory 602, the input output device 603, and the network access device, etc., are connected via the bus.
The Processor 601 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center for the file storage device, with various interfaces and lines connecting the various parts of the overall device.
The memory 602 may be used for storing computer programs and/or modules, and the processor 601 may implement various functions of the computer apparatus by executing or executing the computer programs and/or modules stored in the memory 602 and calling data stored in the memory 602. The memory 602 may generally include a program storage area that may store an operating system, applications required for at least one function (e.g., and the like; and a data storage area that may store data created from use of the file storage device, and the like.
The processor 601, when executing the computer program stored in the memory 602, may specifically implement the following functions:
determining a file to be stored, wherein the file to be stored is a file which is requested by a client and stored in a file storage system;
distributing corresponding target directory servers for files to be stored in a directory server set, wherein the directory server set comprises different directory servers;
the method comprises the steps of sending an acquired file to be stored to a target storage server for storage through a target directory server, and establishing index data between the file to be stored and the target storage server, wherein the target storage server is distributed from a storage server set by the target server, and the storage server set comprises different storage servers.
The present application further provides a file storage system, which includes a directory server set and a storage server set, where the target server set includes different directory servers, and the storage server set includes different storage servers, and the system is configured to execute the file storage method in any embodiment corresponding to fig. 2 to 4, and the description of the hardware structures of the target server and the storage server may refer to the description contents of the hardware structure of the file storage device in the embodiment corresponding to fig. 6.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the file storage apparatus, the device and the corresponding units thereof described above may refer to the description of the file storage method in any embodiment corresponding to fig. 2 to fig. 4, and are not described herein again in detail.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
Therefore, the present application provides a computer-readable storage medium, where a plurality of instructions are stored, where the instructions can be loaded by a processor to execute steps in the file storage method in any embodiment corresponding to fig. 2 to 4 in the present application, and specific operations may refer to descriptions of the file storage method in any embodiment corresponding to fig. 2 to 4, which are not described herein again.
Wherein the computer-readable storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the computer-readable storage medium can execute the steps in the file storage method in any embodiment corresponding to fig. 2 to 4 in the present application, the beneficial effects that can be achieved by the file storage method in any embodiment corresponding to fig. 2 to 4 in the present application can be achieved, for details, see the foregoing description, and are not repeated herein.
The above detailed description is provided for the file storage method, apparatus, device and computer-readable storage medium provided in the present application, and specific examples are applied herein to illustrate the principles and embodiments of the present application, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A method of file storage, the method comprising:
determining a file to be stored, wherein the file to be stored is a file requested by a client to be stored in the file storage system;
distributing a corresponding target directory server for the file to be stored in a directory server set, wherein the directory server set comprises different directory servers;
and sending the acquired file to be stored to a target storage server for storage through the target directory server, and establishing index data between the file to be stored and the target storage server, wherein the target storage server is distributed from a storage server set by the target server, and the storage server set comprises different storage servers.
2. The method according to claim 1, wherein the allocating a corresponding target directory server for the file to be stored in the directory server set comprises:
evaluating the network transmission quality between the different directory servers and the client respectively through the transmission test of the test data packet;
and selecting the directory server corresponding to the optimal network transmission quality as the target directory server.
3. The method according to claim 1, wherein the target storage server includes a plurality of sub-target storage servers, and the sending the acquired file to be stored to the target storage server for storage by the target directory server and establishing index data between the file to be stored and the target storage server includes:
fragmenting the acquired file to be stored through the target directory server to obtain a plurality of fragmented files;
respectively sending the fragment files to a plurality of sub-target storage servers for distributed storage;
and establishing index data between the plurality of fragment files and the plurality of sub-target storage servers.
4. The method of claim 1, further comprising:
and synchronizing the index data between the file to be stored and the target storage server to other directory servers in the directory server set through the target directory server.
5. The method according to claim 1, wherein the sending the acquired file to be stored to a target storage server for storage by the target directory server comprises:
receiving the file to be stored uploaded by the client by the directory server;
and sending the file to be stored to the target storage server for storage through the target server.
6. The method of claim 1, wherein the file storage system is built based on a blockchain platform, and the different directory servers and the different storage servers are all different blockchain link point devices in the blockchain platform.
7. The method of claim 1, further comprising:
receiving a downloading request, wherein the downloading request is used for requesting to download the file to be stored from the file storage system;
determining a target storage server for storing the file to be stored according to the index data;
and providing the downloading service of the file to be stored to the client initiating the downloading request through the target storage server.
8. A file storage device, the method comprising:
the file storage system comprises a determining unit, a storage unit and a storage unit, wherein the determining unit is used for determining a file to be stored, and the file to be stored is a file which is requested by a client and stored in the file storage system;
the distribution unit is used for distributing a corresponding target directory server for the file to be stored in a directory server set, wherein the directory server set comprises different directory servers;
the sending unit is used for sending the obtained file to be stored to a target storage server for storage through the target directory server, wherein the target storage server is distributed by the target server from a storage server set, and the storage server set comprises different storage servers;
and the establishing unit is used for establishing index data between the file to be stored and the target storage server.
9. A file storage system comprising a set of directory servers and a set of storage servers, the set of target servers comprising different directory servers, the set of storage servers comprising different storage servers, the system being configured to perform the method of any of claims 1 to 7.
10. A computer-readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011596322.9A CN112650721A (en) | 2020-12-29 | 2020-12-29 | File storage method, device, system and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011596322.9A CN112650721A (en) | 2020-12-29 | 2020-12-29 | File storage method, device, system and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112650721A true CN112650721A (en) | 2021-04-13 |
Family
ID=75363830
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011596322.9A Pending CN112650721A (en) | 2020-12-29 | 2020-12-29 | File storage method, device, system and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650721A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220644A (en) * | 2021-05-28 | 2021-08-06 | 北京微纳星空科技有限公司 | File processing method, device, equipment and storage medium |
| CN113259449A (en) * | 2021-05-27 | 2021-08-13 | 四块科技(深圳)有限公司 | Distributed storage method, device, equipment and storage medium |
| CN113656348A (en) * | 2021-08-25 | 2021-11-16 | 北京微纳星空科技有限公司 | Distributed cluster management system and method for satellite application big data |
| CN113961908A (en) * | 2021-12-23 | 2022-01-21 | 杭州链城数字科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN117194439A (en) * | 2023-11-07 | 2023-12-08 | 杭州优云科技有限公司 | Method for creating resource storage system, electronic equipment and storage medium |
| CN117215502A (en) * | 2023-11-09 | 2023-12-12 | 深圳市活力天汇科技股份有限公司 | Storage method and storage system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1996843A (en) * | 2005-12-26 | 2007-07-11 | 北大方正集团有限公司 | Light distributed file storage system and file upload method |
| CN103002027A (en) * | 2012-11-26 | 2013-03-27 | 中国科学院高能物理研究所 | System and method for data storage on basis of key-value pair system tree-shaped directory achieving structure |
| JP2013105289A (en) * | 2011-11-14 | 2013-05-30 | Hitachi Solutions Ltd | Delta indexing method for hierarchy file storage |
| CN103888499A (en) * | 2012-12-21 | 2014-06-25 | 中国科学院深圳先进技术研究院 | Distributed object processing method and system |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN111049884A (en) * | 2019-11-18 | 2020-04-21 | 武汉方始科技有限公司 | Distributed large file storage system and file uploading and downloading method |
| CN111258958A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
-
2020
- 2020-12-29 CN CN202011596322.9A patent/CN112650721A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1996843A (en) * | 2005-12-26 | 2007-07-11 | 北大方正集团有限公司 | Light distributed file storage system and file upload method |
| JP2013105289A (en) * | 2011-11-14 | 2013-05-30 | Hitachi Solutions Ltd | Delta indexing method for hierarchy file storage |
| CN103002027A (en) * | 2012-11-26 | 2013-03-27 | 中国科学院高能物理研究所 | System and method for data storage on basis of key-value pair system tree-shaped directory achieving structure |
| CN103888499A (en) * | 2012-12-21 | 2014-06-25 | 中国科学院深圳先进技术研究院 | Distributed object processing method and system |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN111049884A (en) * | 2019-11-18 | 2020-04-21 | 武汉方始科技有限公司 | Distributed large file storage system and file uploading and downloading method |
| CN111258958A (en) * | 2020-01-10 | 2020-06-09 | 北京猎豹移动科技有限公司 | Data acquisition method, data providing method and device |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113259449A (en) * | 2021-05-27 | 2021-08-13 | 四块科技(深圳)有限公司 | Distributed storage method, device, equipment and storage medium |
| CN113220644A (en) * | 2021-05-28 | 2021-08-06 | 北京微纳星空科技有限公司 | File processing method, device, equipment and storage medium |
| CN113656348A (en) * | 2021-08-25 | 2021-11-16 | 北京微纳星空科技有限公司 | Distributed cluster management system and method for satellite application big data |
| CN113961908A (en) * | 2021-12-23 | 2022-01-21 | 杭州链城数字科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN117194439A (en) * | 2023-11-07 | 2023-12-08 | 杭州优云科技有限公司 | Method for creating resource storage system, electronic equipment and storage medium |
| CN117194439B (en) * | 2023-11-07 | 2024-03-22 | 杭州优云科技有限公司 | Method for creating resource storage system, electronic equipment and storage medium |
| CN117215502A (en) * | 2023-11-09 | 2023-12-12 | 深圳市活力天汇科技股份有限公司 | Storage method and storage system |
| CN117215502B (en) * | 2023-11-09 | 2024-03-26 | 深圳市活力天汇科技股份有限公司 | Storage method and storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112650721A (en) | File storage method, device, system and equipment | |
| US11297140B2 (en) | Point of presence based data uploading | |
| US10484465B2 (en) | Combining stateless and stateful server load balancing | |
| EP3296870B1 (en) | Cdn-based content management system | |
| WO2022141021A1 (en) | File storage method, apparatus, system and device | |
| CN110263035A (en) | Data storage, querying method and device and electronic equipment based on block chain | |
| US10798205B2 (en) | Cache system for live broadcast streaming | |
| EP2656552B1 (en) | Third party initiation of communications between remote parties | |
| WO2020253634A1 (en) | Data processing method, system and device | |
| US10129358B2 (en) | Partitioned serialized caching and delivery of large files | |
| CN113517985B (en) | File data processing method and device, electronic equipment and computer readable medium | |
| WO2016180188A1 (en) | Distributed link establishment method, apparatus and system | |
| US11444998B2 (en) | Bit rate reduction processing method for data file, and server | |
| US10122539B2 (en) | URL replication for live broadcast streaming | |
| CN116471586A (en) | Data processing method, device and readable storage medium | |
| CN106790610B (en) | Cloud system message distribution method, device and system | |
| CN112653539B (en) | Storage method, device and equipment for data to be stored | |
| US20100057914A1 (en) | Method, apparatus and system for scheduling contents | |
| CN114036234A (en) | Block chain data storage system and method | |
| US20190052649A1 (en) | Detecting malware on spdy connections | |
| CN110784775A (en) | Video fragment caching method and device and video-on-demand system | |
| CN111416852A (en) | Method for session synchronization among multiple load balancers and load balancer | |
| CN111416851A (en) | Method for session synchronization among multiple load balancers and load balancer | |
| CN112671905B (en) | Service scheduling method, device and system | |
| US20170126571A1 (en) | Method and apparatus for controlling network bandwidth |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210413 |
|
| RJ01 | Rejection of invention patent application after publication |