CN113273163A

CN113273163A - File uploading method, file downloading method and file management device

Info

Publication number: CN113273163A
Application number: CN202080007587.2A
Authority: CN
Inventors: 不公告发明人
Original assignee: DeepRoute AI Ltd
Current assignee: DeepRoute AI Ltd
Priority date: 2020-05-26
Filing date: 2020-05-26
Publication date: 2021-08-17
Also published as: WO2021237467A1

Abstract

The application relates to a file uploading method, a file downloading method and a file management device, wherein the file uploading method comprises the following steps: acquiring a file to be uploaded; dividing the file to be uploaded into at least one block file; uploading the at least one block file to a storage server; receiving at least one storage address corresponding to the at least one block file returned by the storage server; and storing meta information of the file; the meta information includes a first file identifier of a file stored in association, the at least one storage address, and an arrangement order of the at least one block file corresponding to the at least one storage address in the file before the division.

Description

File uploading method, file downloading method and file management device

Technical Field

The present application relates to the field of data storage technologies, and in particular, to a file uploading method, a file downloading method, and a file management apparatus.

Background

At present, there are many Storage service providers providing data Storage services for enterprise users or personal users, and the Storage servers of these Storage service providers may provide access to Object Storage services, such as an Object-Based Storage System (Object-Based Storage System), for the enterprise users to store mobile applications, large websites, picture sharing or hot videos and videos, or to perform low-frequency access Storage and archive Storage, or for the personal users to store files, and so on. The service can provide file flat storage and Content Delivery Network (CDN) resources to improve the loading speed of users on static resources. A user uploads a file to a storage pool of a storage server through a back-end program of a user terminal, so that a Uniform Resource Locator (URL) returned by the storage server can be obtained, the address is embedded into a webpage or data returned by an Application Programming Interface (API), and the user can download the previously uploaded file by means of the URL. However, in the related art, the file stored in the storage server is easily stolen, and the security of file storage is low.

Disclosure of Invention

Based on this, it is necessary to provide a file uploading method, a file downloading method, and a file management apparatus.

A file uploading method comprises the following steps: acquiring a file to be uploaded; dividing a file to be uploaded into at least one block file; uploading at least one block file to a storage server; receiving at least one storage address corresponding to at least one block file returned by a storage server; and storing meta information of the file; the meta information comprises a first file identification of the file stored in an associated mode, at least one storage address and information of the arrangement sequence of at least one block file corresponding to the at least one storage address in the file division front.

A method of file download, comprising: acquiring a first file identifier of a file to be downloaded; acquiring meta information of the file based on the first file identifier; the meta-information comprises a first file identifier of the file, at least one storage address and information of the arrangement sequence of at least one block file corresponding to the at least one storage address in the file before the file is divided; downloading at least one block file from a storage server corresponding to at least one storage address by using at least one storage address; and restoring the at least one block file into a complete file and returning the complete file based on the information of the arrangement sequence of the at least one block file.

A file management device is in communication connection with a storage server; the file management apparatus includes a processor and an information storage. The processor is used for executing the file uploading method and the file downloading method.

According to the file uploading method, the file downloading method and the file management device, when the file is uploaded, the file to be uploaded is divided into at least one block file, the at least one block file is stored in the storage server respectively, when the file is downloaded, the at least one block file is downloaded from the storage server, and the at least one block file is combined according to the dividing sequence of the locally recorded block files to obtain the restored file. Therefore, the files stored in the storage server are incomplete blocked files, and information such as the arrangement sequence of the blocked files is difficult to obtain from the storage server, so that the complete storage files are difficult to restore from the storage server, and the safety of the files stored in the storage server is effectively improved.

Drawings

FIG. 1 is a diagram of an application environment of a file upload method and a file download method in one embodiment;

fig. 2 is a diagram of an application environment of a file uploading method and a file downloading method in another embodiment;

FIG. 3 is a flowchart illustrating a file upload method according to an embodiment;

FIG. 4 is a flowchart illustrating a file upload method according to an embodiment;

FIG. 5 is a flowchart illustrating a file upload method according to an embodiment;

FIG. 6 is a flowchart illustrating a file upload method according to an embodiment;

FIG. 7 is a diagram illustrating an exemplary information storage device;

FIG. 8 is a flowchart illustrating a file download method according to an embodiment;

FIG. 9 is a flowchart illustrating a file download method according to an embodiment;

FIG. 10 is a diagram illustrating a file upload method and a file download method, in accordance with an embodiment;

FIG. 11 is a block diagram showing the configuration of a file upload apparatus according to an embodiment;

FIG. 12 is a block diagram showing the configuration of a file downloading apparatus according to an embodiment;

FIG. 13 is a block diagram showing the configuration of a file management apparatus according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The file uploading method and the file downloading method provided by the application can be applied to the application environment shown in fig. 1. Where user device 102 communicates with intermediate server 104 over a network, intermediate server 104 communicates with storage server 106 over a network. The storage server 106 is typically a third party server. The user device 102 may be an enterprise user device or a personal user terminal, and the enterprise user device may be an enterprise server and/or an enterprise user terminal. The personal user terminal and the enterprise user terminal may be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like. The storage service provider provides a storage service via the storage server 106, and a storage pool for storing files is provided in the storage server 106.

The file uploading method and the file downloading method provided by the application can be executed by a file management device, the file management device can store a back-end program, the back-end program can be deployed on the intermediate server 104, and the file uploading method and the file downloading method of the application are executed when the back-end program runs. In other embodiments, the backend program may also be deployed partially in the intermediate server 104 and partially in the user device 102 by the intermediate server 104, or may also be deployed fully in the user device 102 by the intermediate server 104. Accordingly, a part of the steps of the file uploading method and the file downloading method of the present application may be executed in the intermediate server 104, another part of the steps may be executed in the backend program of the user equipment 102, or all the steps may be executed in the backend program of the user equipment 102.

In one embodiment, the file management apparatus may include an information storage 110, and the information storage 110 may include an internal memory and a nonvolatile storage medium. The non-volatile storage medium may have the back-end program stored thereon. The nonvolatile storage medium may further store a database, where the database may be used to store index information of the uploaded file, where the index information may include meta information of the file, or may obtain the meta information of the file through the index information, for example, the index information may also include a storage address of the associated meta information and a first file identifier. The information store 110 may be located in the user device 102 or in the intermediate server 104.

In one embodiment, storage services provided by a plurality of different storage service providers may be utilized in a comprehensive manner, and accordingly, as shown in fig. 2, the intermediate server 104 may communicate with a plurality of storage servers 106 corresponding to the plurality of storage service providers via a network, and the file management apparatus may select to store the divided at least one block file into the plurality of different storage servers 106, respectively. The intermediate server 104 and the storage server 106 may each be implemented as a stand-alone server or as a server cluster of multiple servers.

In an embodiment of the application, the index information of the file can be stored in a Redis database, and a file system directory structure can be realized in the database by using a Hash and Set data structure. According to the actual situation, whether the file meta information is cached in the Redis database or not can be selected, or the URL of the webpage where the meta information is located is recorded. Meanwhile, information such as file names, actual file lengths, file creating and modifying time and the like is stored in a Redis database, so that the query of the information is accelerated.

In the above embodiment, the Redis database communicates with the TCP protocol, so that multiple framework instances can be configured to connect to the same Redis database instance, thereby easily implementing file index synchronization. And the Redis database supports atomic operation, so that various competition conditions can be effectively avoided. Meanwhile, the Redis database supports a master-slave architecture, is convenient to expand and can provide high-availability guarantee. In addition, the RDB function and the AOF function of the Redis database can be used for realizing persistence, so that the loss of the file index is prevented to the greatest extent, and the integrity of the frame data is ensured.

In an embodiment of the present application, a directory is established in other file systems as a root directory of the file system, and all operations in which the granularity of the program interface is larger than the file itself read-write can be completed by a method of calling a POSIX interface by the system, that is, a transparent proxy of a directory structure is realized. And then storing the meta information of the frame file in files of corresponding directory structures of other file systems, and forwarding the request to the bottom layer of the frame abstract file for processing when operating the file data.

In the above embodiment, the file system transparent proxy method uses the existing mature file system to host the file index of the framework, and has higher stability and security. And a part of data in the meta information can be directly kept consistent with ATTR attributes supported in the existing file system, such as the creation time ctime and the modification time mtime. For attributes that cannot be kept consistent, such as the true size of the file, the output can be simulated by reading the meta-information at the transparent proxy layer.

In an embodiment of the present application, as shown in fig. 3, a file uploading method is provided, where the file uploading method is applied to the intermediate server 104 in fig. 1, and the example where a backend program of the file management apparatus of the present application is deployed on the intermediate server 104 is described, including the following steps S302 to S310.

Step S302, a file to be uploaded is obtained.

Specifically, the intermediate server 104 receives a file upload request sent by the user equipment 102, where the file upload request carries a file to be uploaded. The intermediate server 104 parses the file upload request to obtain the file to be uploaded.

In an embodiment of the application, the file management apparatus may provide both the HTTP API interface and the POSIX interface. Taking the user equipment 102 as an example to perform a file uploading operation by using an HTTP API interface, as shown in fig. 10, a user directly initiates a file uploading request (POST/PUT request for short) with a POST/PUT method type to the intermediate server 104 by using a Web front end of a browser or a dedicated client or by other means, where a request body of a main body part of the POST/PUT request carries a file to be uploaded. Thus, in this step, the intermediate server 104 may receive a POST/PUT request of the user equipment 102 carrying the file to be uploaded.

Step S304, the file to be uploaded is divided into at least one block file.

Specifically, the intermediate server 104 may divide the file to be uploaded into one or more block files according to a predetermined file division rule. In one embodiment, as shown in FIG. 10, the intermediate server 104 instantiates an abstract file object that may provide a number of methods for different developers to invoke for use, including, for example, flush, truncate, and write methods available in write mode, and locate, read, seek, and tell methods available in read mode. And setting the abstract file object to be in a write (write) mode, and transmitting the file data stream of the file obtained by analyzing in the step S802 into a write method, wherein the file data stream is cut by the write method according to the pre-configured block size, so that a multi-segment block file data stream is obtained. The multi-segment block file data stream represents at least one corresponding block file.

Step S306, uploading at least one block file to a storage server.

The storage server 106 is provided with an upload interface, and the intermediate server 104 may respectively send a plurality of file upload requests to the storage server 106 through the upload interface provided by the storage server 106 for at least one partitioned file, where each file upload request carries one data stream of the plurality of data streams, so as to upload the plurality of data streams to the specified storage server 106.

When at least one block file is uploaded to the storage server, a multithreading or asynchronous mode can be used for uploading multiple segments of data streams, namely at least one block file to the storage server 106 independently through different coroutines or threads, so that the macroscopically synchronous uploading of at least one block file is realized, bandwidth resources are fully utilized, and the transmission speed is increased.

Step S308, at least one storage address corresponding to at least one block file returned by the storage server is received.

When receiving any one of the block files uploaded by the intermediate server 104, the storage server 106 stores the block file in the storage pool of the storage server 106, and sends the corresponding storage address to the intermediate server 104. The storage address is used to indicate a storage location of the block file in the storage server 106. In one embodiment, the storage address may be a Uniform Resource Locator (URL).

Step S310, storing the meta information of the file, wherein the meta information of the file comprises a first file identifier of the file stored in a related manner, at least one storage address, and an arrangement sequence of at least one block file corresponding to the at least one storage address before the file is divided.

The meta information of the file may record information related to various processing performed on the file during the file uploading process, such as disguising, dividing, uploading, and the like. In addition, the meta information may also include other information related to the file, for example, information such as a file name of the file, a MIME type of the file, an actual file length, a modification time of the file, and the like, carried in a file upload request of the file may be included. Therefore, when a file is uploaded, the file data stream can be extracted from the request main body of the file uploading request, the file data stream is divided into a plurality of data streams, and the plurality of data streams are uploaded. While other file-related information extracted in the request is stored in the meta-information.

The first file identifier is information that uniquely identifies the file, and when two files have the same first file identifier, the two files can be considered to be the same file. In an embodiment, the first file identifier may be a check value of the file, such as a file fingerprint, and the contents of the file may be compared with higher accuracy by calculating the file fingerprint. In other embodiments, the first file identifier may also be other information related to the file, such as a storage path of the file in the information storage, etc., according to the requirement for the file identification precision.

According to the file uploading method, the file to be uploaded is divided into the plurality of block files, and the plurality of block files are respectively stored in the storage server, so that the file stored in the storage server is an incomplete block file, and information such as the arrangement sequence of the block files is difficult to obtain from the storage server, so that the complete storage file is difficult to restore from the storage server, and the safety of the file stored in the storage server is effectively improved.

In one embodiment, the meta information of the storage file of step S310 includes: meta information of the file is stored in an information storage.

In the embodiment, the meta information of the file is stored in the information storage, so that the meta information of the file cannot be acquired from the storage server, the uploaded original file is difficult to restore, and the security of the file stored in the file storage is further improved.

In one embodiment, the file meta information of the present application can also be uploaded to the storage server. The meta information of the storage file of the above step S310 includes: uploading the meta information of the file to a storage server; receiving a storage address of the meta information returned by the storage server; and storing the storage address of the meta information in the information storage in association with the first file identification of the file. The storage address of the meta information may also be a URL.

In the above embodiment, by uploading the meta information of the file to the storage server, the information storage of the information storage may only store the storage address of the meta information of the file and the information of the first file identifier, so as to reduce the data storage burden of the information storage and save the data storage capacity of the system.

Further, in an embodiment, the dividing the file to be uploaded into the plurality of block files in step S304 includes: based on a preset size, segmenting the file to be uploaded to obtain a first number of block files with the preset size; the value of the preset size is smaller than or equal to the upper limit value of the file size allowed to be stored by the storage server, and the first quantity is a value rounded by the quotient of the size of the file to be stored and the preset size; and after the first number of block files with the preset size are obtained through division, if the file has a residual part, distributing the residual part of the file into one block file.

In one embodiment, when there are a plurality of storage servers, the chunk sizes of at least one chunk file may be the same as each other or different from each other, as long as the predetermined size of each chunk file is less than or equal to the upper limit value of the file size allowed to be stored by the storage server to be stored for which the chunk file corresponds.

According to the scheme of the embodiment of the application, the file to be stored is divided into the partitioned files with the size smaller than or equal to the upper limit value of the size of the file allowed to be stored by the storage server, so that the limitation of the storage service provider on the size of the stored file is met, and the file with any size can be uploaded.

The meta information of the file to be stored is generally small and can meet the file size limit of the storage server, but in some cases, the meta information is large and may exceed the upper limit value of the file size allowed to be stored by the storage server. In an embodiment, the meta-information may be further divided so that the size of each piece of partitioned meta-information is smaller than the upper limit value, then the plurality of pieces of partitioned meta-information obtained by the division are uploaded to the storage server respectively, and the storage address of the piece of partitioned meta-information returned by the storage server is received and stored, so as to satisfy the limitation of the storage service provider on the size of the storage file. When downloading the file, the storage address of the block meta-information can be used to obtain the meta-information of the file in the same way.

In one embodiment, as shown in fig. 4, after step S304 and before step S306, the file uploading method further includes: s404, selecting an encoder with a target file format, and respectively disguising at least one block file into the target file format by using the encoder; the target file format comprises a file format which is allowed to be stored by the storage server; accordingly, the meta information stored in step S310 further includes information of at least one encoder for corresponding use of the disguised at least one block file.

The encoder is a module for disguising an input file with any file format into a file with a specified target file format and then outputting the file. The target file format of the encoder is a target file format in which the file is disguised. The encoder may perform a process of adding a header and an end of a file or some other change process to each received data stream to disguise the block files represented by the data streams into a block file in the target file format.

In one embodiment, at least one of the block files may be disguised as a plurality of block files of the same file format, at this time, one encoder may be selected from at least one encoder having the same target file format to disguise the at least one block file, and the setting of the at least one encoder may prevent a single encoder from failing to perform disguising processing. Or, at least one of the block files may also be disguised as a plurality of block files of different file formats, and at this time, at least one of the encoders having different target file formats may be selected to disguise at least one of the block files, respectively.

In one embodiment, as shown in fig. 10, after the multiple segments of the segmented file data streams are cut by the write method, each of the cut segmented file data streams may be respectively directed to a corresponding encoder having a target file format, and the encoder may perform a process of adding a file header and a file trailer or some other change process on each of the received segmented file data streams to masquerade the segmented files represented by the segmented file data streams into the segmented files having the target file format.

In the embodiment, at least one partitioned file is disguised to have a target file format before the partitioned file is uploaded, the storage server cannot easily obtain the partitioned file before disguising, the requirement of the file format allowed to be stored by the storage server is met, meanwhile, the difficulty of obtaining the original file from the storage server in a restoring mode is increased, and the safety of file storage is further improved.

In one embodiment, as shown in fig. 5, after step S302, the file uploading method further includes: step S502, acquiring a first file identifier of a file to be uploaded; step S504, searching a first file identifier in an information memory and a storage server; step S506, judging whether the first file identification is stored in the information memory and the storage server; when the first file identifier is not stored in the information storage and the storage server, continuing to execute the step S406 of dividing the file to be uploaded into at least one block file; when the first file identifier is already stored in the information storage or the storage server, step S508 is executed to terminate the uploading operation of the file.

In the above embodiment, when a file needs to be uploaded, the first file identifier of the file to be uploaded is first searched in the information storage and the storage server, and if the stored first file identifier is found in any one of the information storage and the storage server, which indicates that the same file has been uploaded before, the uploading operation of the file is terminated, and only the meta information of the file is updated and stored. Unnecessary storage space and system processing resources occupied by repeated uploading are avoided.

In one embodiment, the first file identifier is a file fingerprint, and the file fingerprint of the file to be uploaded can be calculated based on the data with fixed byte length in the header of the file data stream of the file and the actual file length. As shown in fig. 10, still taking the POST/PUT request as an example, when receiving the POST/PUT request, the intermediate server 104 first parses the request header to obtain the original data Length, i.e. the value of Content-Length in the request header. The data format of the request body, which is specified by the value of Content-Type in the request header, is then obtained. When the value of the Content-Type is multipart/form-data; when Boundary is 3vkqffBXJh (where 3vkqffBXJh is a custom delimiter), the data stream recorded in the request body is directed to a Multipart parser, the parser uses regular matching to separate the data stream contents such as file name, Multipurpose Internet Mail Extensions (MIME) type, file data stream and the like from the data stream, and then the length of the non-file data stream parts such as Boundary objects Boundary, file name, MIME type and the like is subtracted from the original data length to obtain the actual file length. When the value of the Content-Type is not multipart/form-data; when the boundary is 3vkqffBXJh, the request body is considered to be an unprocessed original file data stream, the original data length is the actual file length, and meanwhile, a file name and a Content-Type value can be obtained from a parameter carried in a URL of a request header or a Content-displacement of the request header as a MIME Type, and information such as the actual file length and the file uploading time is obtained for subsequently storing the meta information of the file. A file fingerprint is then calculated based on the header fixed byte length data and the actual file length of the file data stream, wherein the file data stream includes the header fixed byte length data and the remaining data.

When a user uploads a file through a provided POSIX interface, similarly, during uploading, the intermediate server 104 receives a data WRITE request (WRITE request for short) with a WRITE (WRITE) method type initiated by the user through a user space file system (FUSE), and then extracts data content of the file to be uploaded from the WRITE request and calculates a first file identifier.

In one embodiment, in step S508, meta information of the file may be further stored, and of the stored meta information in the file, information related to the block files may be obtained from the meta information corresponding to the matched previously stored first file identifier, and other information related to the file, such as the uploading time, may be obtained from the file uploading request of the file.

In one embodiment, as shown in fig. 4, after step S304 and before step S404, the file uploading method further includes: step S402, calculating at least one second file identifier corresponding to at least one block file; the meta information stored in step S310 further includes at least one second file identifier corresponding to the at least one block file stored in association with the block file.

The second file identification is information for uniquely identifying the file. By calculating and storing the second file identifier of the block file before performing the disguise of the file, the characteristic information of the original block file before disguise can be recorded.

In one embodiment, as shown in fig. 10, the intermediate server 104 may collect a set of information of each chunk file, including a chunk URL, a chunk file fingerprint, a start offset of a data stream of the chunk file before disguising in a data stream of the chunk file after disguising, a data length of the data stream of the chunk file before disguising, and the like, through an abstract file object, to obtain N sets of information corresponding to N chunk files. And arranging the N groups of information according to the arrangement sequence of the block files before file segmentation to form a block information sequence. Specifically, a start offset of the block file may be used as a key, and other information, such as a block URL, a block file fingerprint, a data length, and the like, may be used as values, so as to form a key-value pair of the block file to be inserted into a dictionary (ditt). Under the condition that at least one block file is uploaded by multiple threads, after all threads are uploaded, the key value pairs corresponding to all the block files are sequenced in the dictionary according to the sequence of the block files before file segmentation, and the sequenced dictionary is a block information sequence. And then serializing the block information sequence and information such as file name, file modification time, MIME type, file fingerprint, uploading time, used encoder and the like of the file by using JSON and other formats so as to obtain the meta information of the file. The extension of the file name has a one-to-one correspondence relationship with the MIME type, so that the stored meta information of the file may not include the MIME type, and when the downloaded block file needs to be restored to obtain the original file, the corresponding MIME type can be derived from the extension of the file name.

In addition to searching for the entire file before uploading the file, the present application may also search for the block file before uploading the file. In one embodiment, as shown in fig. 6, after step S402 and before step S306, the file uploading method further includes: step S602, each of the at least one second file identifier is respectively searched in the information storage and the storage server.

Accordingly, steps S306 and S308 include: step S604, taking a second file identifier which is not stored in the information memory and the storage server in at least one second file identifier as a target second file identifier, and uploading a blocked file corresponding to the target second file identifier to the storage server; and step S606, receiving the storage address of the block file corresponding to the target second file identifier sent by the storage server.

In this embodiment, the second file identifier may be a check value of the file, for example, a file fingerprint of the file, and the contents of the file may be compared with higher accuracy by calculating the file fingerprint of the file. In the transmission process of the data stream corresponding to the file, hash algorithms such as SHA-1 and the like can be used for iterative computation of the check value of the file.

In the above embodiment, the file to be uploaded is divided into the block files, and then, with the block files as a unit, whether the same file has been uploaded previously for each block file is searched in the information storage and the storage server of the intermediate server, and when the block files are uploaded, only the block files that have not been uploaded previously need to be uploaded, and the block files that have been uploaded previously do not need to be repeatedly uploaded, so that the file storage capacity and the system processing resources are further saved. In one embodiment, as shown in fig. 6, after step S602 and before step S310, the file uploading method further includes: step S608, the second file identifier already stored in the information storage or the storage server in the at least one second file identifier is used as the stored second file identifier, and the storage address of the block file corresponding to the stored second file identifier is obtained. Accordingly, the meta information stored in step S310 includes the first file identifier of the file stored in association, the target second file identifier, the stored second file identifier, the storage address of the block file corresponding to the target second file identifier, the storage address of the block file corresponding to the stored second file identifier, and the information of the arrangement order in the file before the block file corresponding to the target second file identifier and the block file corresponding to the stored second file identifier are divided.

In the above embodiment, for at least one block file obtained by segmenting one file to be uploaded, after the block file which has not been uploaded previously is uploaded, the meta information of the block file which has been uploaded previously and the meta information of the currently uploaded block file can be respectively combined to obtain the complete meta information of the file to be uploaded, so that the corresponding original file can be obtained by downloading based on the meta information of the file during downloading.

In one embodiment, as shown in fig. 7, the information storage 110 of the present application may store a database including at least one public database 111 and a plurality of local databases 112. Wherein, each user device 102 uses one local database 112, the local database 112 is dedicated to store the index information of the corresponding user device 102 for the individual user, the local database 112 may be configured on the personal user terminal, and for the enterprise user, the local database may be configured on the enterprise server or the enterprise user terminal of the enterprise. The common database 111 may be used to store index information for sharable files uploaded by multiple user devices 102. Therefore, the index information of the file can be shared among the instances, so that when new index information is added to any instance using the framework, the index information can be inquired by other instances, and the file can be shared to all the instances.

In one embodiment, the number of the storage servers is multiple; the number of the block files is multiple; the uploading the at least one block file to the storage server comprises: uploading the plurality of block files to a plurality of storage servers respectively, so that each storage server in the plurality of storage servers stores a part of the block files in the plurality of block files. Therefore, each storage server only stores a part of the block files of the plurality of block files of the file, but not stores all the block files of the file, so that the complete original file is difficult to restore from a single storage server, and the safety of file storage is improved.

In one embodiment, the number of the storage servers is multiple; the uploading the at least one block file to the storage server comprises: uploading the at least one chunked file to a plurality of the storage servers, respectively, such that the at least one chunked file is repeatedly stored in at least two of the plurality of storage servers. Therefore, each block file is backed up and stored in more than two storage servers, and when the block file is downloaded, if the correct block file cannot be downloaded from a certain storage server, the block file can be downloaded from a standby storage server, so that the reliability of file downloading is ensured.

Further, in an embodiment, when there are more than three storage servers, the storing at least one block file to a storage server may further include, in combination with the two embodiments: the plurality of block files are stored to a plurality of storage servers, respectively, such that each of the plurality of storage servers stores a portion of the block files of the plurality of block files, and each of the plurality of block files is stored in at least two of the plurality of storage servers. Therefore, the reliability of file downloading can be improved while the file storage safety is ensured.

In one embodiment, the file uploading method may further include a step of specifying that data of an arbitrary byte range of an uploaded file is updated. Specifically, when receiving a file update request for updating a specified byte range of a file, the intermediate server 104 finds meta information of an old file uploaded previously based on a first file identifier, then determines one or more uploaded block files partially or completely covered by the file update range based on the file update range specified in the file update request, segments and disguises a new file to be uploaded in the file update request based on the start position and the end position of each block file before disguising corresponding to the uploaded block files, then replaces the old block file stored in the storage server with the segmented and disguised new block file, and updates meta information of the replaced block file, i.e., can update data of any byte range of the uploaded file. Further, when the start position and the end position of the file update range requesting updating are not completely aligned with the start position and the end position of the previously uploaded old block files, that is, an offset difference exists between the start position and the end position, the corresponding start block file and/or end block file of the downloaded plurality of block files can be segmented according to the start position and the end position of the specified byte range, the part outside the specified byte range is removed, and one or more block files within the specified byte range are reserved and restored, so that the files conforming to the specified byte range are accurately downloaded. The start position and the end position of each block file may be calculated according to the data length of the block file described in the meta information of each block file. The file update Range can also be specified by using HTTP Range, and the Web server can parse a POST/PUT request initiated by a client to obtain Range data in a request header.

In one embodiment, as shown in fig. 8, the present application provides a file download method. Taking the file downloading method applied to the intermediate server 104 in fig. 1 and the deployment of the backend program of the present application on the intermediate server 104 as an example for explanation, the following steps S802 to S810 may be included.

Step S802, a first file identifier of a file to be downloaded is obtained.

Specifically, the intermediate server 104 may receive a file download request sent by the user equipment 102, where the file download request carries a first file identifier of a file to be downloaded. The intermediate server 104 parses the file downloading request to obtain a first file identifier of the file to be downloaded.

Step S804, based on the first file identification, obtaining the meta information of the file; the meta information of the file comprises a first file identifier of the file, at least one storage address and information of the arrangement sequence of at least one block file corresponding to the at least one storage address in the file before the file is divided.

In this step, the intermediate server 104 finds the meta information of the file based on the first file identifier of the file.

Step S806, downloading at least one block file from a storage server corresponding to the at least one storage address by using the at least one storage address.

In this step, the intermediate server 104 generates a plurality of corresponding file download requests by using at least one storage address, respectively, and sends the plurality of file download requests to the corresponding one or more storage servers 106, respectively, and downloads a corresponding block of data from the storage server corresponding to each storage address, thereby obtaining at least one block file corresponding to the at least one storage address one to one.

In one embodiment, there are multiple storage servers, and the intermediate server 104 needs to know which storage server corresponds to each storage address before sending the file download request corresponding to the storage address. When the storage address returned by the storage server carries the information of the storage server, the intermediate server 104 may read the information of the corresponding storage server from the storage address.

Step S808, restoring the at least one block file into a complete file and returning the complete file based on the information of the arrangement sequence of the at least one block file.

Specifically, the intermediate server 104 may restore the at least one block file to a complete file based on the information of the arrangement order of the at least one block file corresponding to the at least one storage address stored in the meta information before the file is split, and return the complete file to the user equipment 102.

The file downloading method of the present application can restore the downloaded at least one block file to a complete file based on the stored information of the arrangement order of the at least one block file, and return the complete file to the user equipment 102.

In one embodiment, when the meta information of the file is stored in the information storage, the obtaining the meta information of the file based on the first file identifier in step S504 in the file downloading method includes: and searching and acquiring the meta information of the file in the information storage based on the first file identification.

In an embodiment, when the meta information of the file is stored in the storage server, the obtaining the meta information of the file based on the first file identifier in step S804 in the file downloading method includes: searching and acquiring a storage address of the meta information of the file in the information storage based on the first file identifier; and downloading the meta information of the file from a storage server corresponding to the storage address of the meta information using the storage address of the meta information.

In the above embodiment, by uploading the meta information of the file to the storage server, the information storage may only store the storage address of the meta information of the file and the information of the first file identifier, thereby reducing the data storage burden of the information storage and saving the data storage capacity of the system.

In an embodiment, as shown in fig. 9, the meta information obtained in step S804 further includes at least one second file identifier corresponding to at least one block file stored in association; after step S806, and before step S808, the file downloading method further includes: step S904, at least one third file identifier corresponding to the downloaded at least one block file is calculated. Step 906, when the third file identifier of the block file matches the second file identifier of the block file, determining that the block file passes the check. Step 908, when the third file identifier of the blocking file is not matched with the second file identifier of the blocking file, determining that the blocking file is not verified, and re-downloading the blocking file, and replacing the re-downloaded blocking file with the blocking file which is not verified.

The third file identifier and the second file identifier are both information for uniquely identifying the partitioned files, and when two partitioned files have the same second file identifier or third file identifier, the two partitioned files can be considered as the same file. In this embodiment, the second file identifier and the third file identifier may be check values of the block file, for example, block file fingerprints of the block file, and the contents of the block file may be compared with higher accuracy by calculating the block file fingerprints of the block file. In the transmission process of the data stream corresponding to the block file, the check value of the block file can be iteratively calculated by using hash algorithms such as SHA-1 and the like.

In the above-described embodiment, by calculating and storing the second file identification of the block file before performing the file masquerading, the feature information of the original block file before masquerading can be recorded. After the partitioned file before disguise is downloaded and restored, the third file identification of the partitioned file before disguise obtained through downloading is calculated and compared with the second file identification of the partitioned file before disguise obtained through uploading, when all the partitioned files pass verification, the complete file obtained through restoring can be determined to be the same as the original file obtained through uploading, and therefore the integrity and the reliability of the downloaded file can be guaranteed.

Further, in one embodiment, when uploading a file, before disguising the block files in step S404, a redundant portion including an erasure code, such as Reed Solomon code (RS code), may be added to each block file; when downloading the file, when checking each downloaded block file, as an alternative to step 908, if the current block file does not pass the check, the block file may be recovered by using the erasure code of the block file, so as to improve the stability of file downloading. Accordingly, before the complete file is restored in step S808, the redundant portion in the block file needs to be deleted, so that the restored complete file is consistent with the original file.

When downloading a file, at least one partitioned file before disguising needs to be acquired to restore and obtain an original file. In one embodiment, as shown in fig. 9, the meta information acquired in step S804 further includes information of at least one encoder corresponding to at least one block file; after step S806, and before step S904, the file download method further includes: s902, acquiring at least one decoder corresponding to at least one encoder based on the at least one encoder; and using at least one decoder to respectively restore the corresponding at least one block file into at least one block file before disguising.

The decoder is a module for restoring an input file which is disguised as a specified target file format into a file before disguising. The target file format that the decoder has represents that the decoder will restore the file that is disguised as the target file format to its original format. The decoder can extract the part of the blocked file before disguising from each received data stream, and ignore or delete the parts of the file head, the file tail and the like added in the disguising way so as to obtain the blocked file before the disguising way. The encoder and decoder have a one-to-one correspondence. In this embodiment, after obtaining the information of the encoder, the information of the corresponding decoder can be obtained, and the disguised block file is decoded into the block file before disguising by using the information of the corresponding decoder.

As an alternative to step S902, in another embodiment, when each of the at least one block file stored in the storage server is disguised as a block file having a predetermined format, step S806 specifically includes: and downloading the block files before disguising from the disguised block files stored in the storage server corresponding to the storage address by using the position of each block file before disguising in the corresponding disguised block file.

The position of the block file before disguising in the block file after disguising comprises a position interval formed by the starting position and the ending position of the block file before disguising in the block file after disguising. The location section may be determined based on the data length of the block file before masquerading and a start offset of the block file before masquerading in the block file after masquerading. Such information may be recorded in meta information of the file.

In the above embodiment, for the blocked file that is disguised before being uploaded, it is not necessary to download the complete blocked file and then perform the decoding process. In the process of downloading the block files, the position of the block files before disguising in the disguised block files is utilized, data of the position interval occupied by the block files before disguising is designated to be downloaded directly from the disguised block files stored in the storage server, and data at other positions can be downloaded without downloading. Specifically, the HTTP Range request header may be used to specify that data of a specified byte Range of a certain block file is downloaded when the block file is downloaded. Therefore, the partitioned file before disguising can be directly downloaded, the data transmission flow is further saved, and the performance cost of the system is also saved.

Similarly, the characteristics of data in a specified byte Range for downloading a certain block file can be extended to the whole file, and a user can use some existing downloading tools to realize multi-thread downloading and breakpoint transmission by writing a Web server which can also receive and analyze an HTTP Range request header. At this time, the Web server can obtain the aforementioned value capable of extracting the meta information of the file, such as the first file identifier, from the URL PATH or the form part of the GET/POST request initiated by the client, then parse the Range data in the request header, determine and download one or at least one block file corresponding to the file download Range for the file download Range specified in the Range data, and then return the specified downloaded data. When the starting position and the ending position of the specified byte range of the request are not completely aligned with the starting position and the ending position of the stored block files, namely an offset exists between the starting position and the ending position, the corresponding starting block files and/or ending block files in the downloaded plurality of block files can be segmented according to the starting position and the ending position of the specified byte range, the parts outside the specified byte range are removed, one or more block files within the specified byte range are reserved and restored, and therefore the files meeting the specified byte range are accurately downloaded.

In one embodiment, the number of the storage servers is multiple; the number of the block files is multiple; each of the plurality of partitioned files is stored in duplicate on at least two of the plurality of storage servers; the downloading, by using the at least one storage address, the at least one block file from the storage server corresponding to the at least one storage address includes: downloading a corresponding block file from a storage server corresponding to a first storage address according to the first storage address in a plurality of storage addresses corresponding to each block file in a plurality of block files; when downloading of the corresponding block file from the storage server corresponding to the first storage address by using the first storage address fails, downloading the corresponding block file from the storage server corresponding to the second storage address by using a second storage address of the plurality of storage addresses corresponding to the block file.

In one embodiment, when the download using the second storage address also fails, if there are other storage addresses, the downloading of the block file using the other storage addresses may be continued by exchanging until the desired block file is downloaded. Therefore, the reliability of file downloading can be effectively improved.

It should be understood that although the various steps in the flow charts of fig. 3-9 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 3-9 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 11, there is provided a file uploading apparatus 1100, including: an upload file acquisition module 1101, a file division module 1102, a file upload module 1103, a storage address reception module 1104 and a meta information storage module 1105,

an upload file acquiring module 1101 is configured to acquire a file to be uploaded.

The file dividing module 1102 is configured to divide a file to be uploaded into at least one block file.

A file uploading module 1103, configured to upload at least one block file to the storage server.

A storage address receiving module 1104, configured to receive at least one storage address corresponding to at least one block file returned by the storage server.

A meta information storing module 1105, configured to store meta information of the file.

In one embodiment, as shown in fig. 12, there is provided a file downloading apparatus 1200 including: a file identifier obtaining module 1201, a meta information obtaining module 1202, a file downloading module 1203, and a file restoring module 1204, where:

the file identifier obtaining module 1201 is configured to obtain a first file identifier of a file to be downloaded.

A meta information obtaining module 1202, configured to obtain meta information of the file based on the first file identifier. The meta information of the file comprises a first file identifier of the file, at least one storage address and information of the arrangement sequence of at least one block file corresponding to the at least one storage address in the file before segmentation.

A file downloading module 1203, configured to download, by using the at least one storage address, the at least one blocking file from a storage server corresponding to the at least one storage address.

The file restoring module 1204 is configured to restore the at least one block file to a complete file and return the complete file based on the information of the arrangement order of the at least one block file.

In one embodiment, as shown in fig. 13, the present application provides a file management apparatus 1300, the file management apparatus 1300 is communicatively connected with a storage server 106; the file management apparatus 1300 includes a processor 1301 and an information storage 110; processor 1301 is configured to:

when receiving a file uploading request, executing: acquiring a file to be uploaded; dividing a file to be uploaded into at least one block file; uploading at least one block file to a storage server; receiving at least one storage address corresponding to at least one block file returned by a storage server; storing meta information of the file;

when receiving a request for downloading a file, executing: acquiring a first file identifier of a file to be downloaded; acquiring meta information of the file based on the first file identifier; downloading at least one block file from a storage server corresponding to at least one storage address by using at least one storage address; and the information of the arrangement sequence of at least one block file, restore at least one block file into a complete file and return the complete file;

the information storage 110 is configured to store meta information of the file or a storage address of the meta information, where the meta information of the file includes a first file identifier of the file, the at least one storage address, and an arrangement order of the at least one block file corresponding to the at least one storage address before the file is divided, where the storage server is configured to store and upload the file.

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method of file upload, the method comprising:

acquiring a file to be uploaded;

dividing the file to be uploaded into at least one block file;

uploading the at least one block file to a storage server;

receiving at least one storage address corresponding to the at least one block file returned by the storage server; and

and storing the meta information of the file, wherein the meta information of the file comprises a first file identifier of the file, the at least one storage address and the arrangement sequence of the at least one block file corresponding to the at least one storage address in the file before the division.

2. The method of claim 1, wherein the storing meta information of the file comprises:

storing meta information of the file in an information store.

3. The method of claim 1, wherein the storing meta information of the file comprises:

uploading the meta information of the file to a storage server;

receiving a storage address of the meta information returned by the storage server;

storing the storage address of the meta-information in an information storage in association with a first file identification of the file.

4. The method of claim 1, wherein the segmenting the file to be uploaded into at least one block file comprises:

based on a preset size, segmenting the file to be uploaded to obtain a first number of block files with the preset size; the value of the preset size is smaller than or equal to the upper limit value of the file size allowed to be stored by the storage server, and the first quantity is a value rounded by the quotient of the size of the file to be stored and the preset size;

and after the first number of block files with the preset size are obtained through division, if the file has a residual part, distributing the residual part of the file into one block file.

5. The method of claim 1, wherein after the segmenting the file to be uploaded into at least one block file and before uploading the at least one block file to a storage server, further comprising:

selecting an encoder having a target file format, using which the at least one block file is respectively disguised as having the target file format;

wherein the target file format comprises a file format which the storage server allows to store;

the stored meta-information of the file further includes information of a corresponding at least one encoder used to disguise the at least one block file.

6. The method of claim 1, wherein the obtaining the file to be uploaded further comprises:

acquiring a first file identifier of the file to be uploaded;

searching the first file identification in an information storage and a storage server; and

when the first file identifier is not stored in the information storage and the storage server, continuing to execute the step of dividing the file to be uploaded into at least one block file;

and when the first file identification is already stored in the information storage or the storage server, the uploading operation of the file is terminated.

7. The method of claim 1, wherein the segmenting the file to be uploaded into at least one block file further comprises:

calculating at least one second file identifier corresponding to the at least one block file;

wherein, the meta-information further includes at least one second file identifier corresponding to the at least one block file stored in association.

8. The method of claim 7, wherein,

after the at least one second file identifier corresponding to the at least one partitioned file is calculated, each second file identifier in the at least one second file identifier is respectively searched in the information storage and the storage server;

the uploading the at least one block file to a storage server, and the receiving at least one storage address corresponding to the at least one block file returned by the storage server includes:

determining that a second file identifier which is not stored in the information storage and the storage server in the at least one second file identifier is used as a target second file identifier, and uploading a blocked file corresponding to the target second file identifier to the storage server;

and receiving the storage address of the block file corresponding to the target second file identifier returned by the storage server.

9. The method of claim 8, further comprising:

and determining that the second file identifier which is already stored in the information memory or the storage server in the at least one second file identifier is used as an already stored second file identifier, and acquiring a storage address of the block file corresponding to the already stored second file identifier.

10. The method of claim 1, wherein the storage server is plural in number; the number of the block files is multiple; the uploading the at least one block file to the storage server comprises:

uploading the plurality of block files to a plurality of storage servers respectively, so that each storage server in the plurality of storage servers stores a part of the block files in the plurality of block files.

11. The method of claim 1, wherein: the number of the storage servers is multiple; the uploading the at least one block file to the storage server comprises:

uploading the at least one chunked file to a plurality of the storage servers, respectively, such that the at least one chunked file is repeatedly stored in at least two of the plurality of storage servers.

12. A method of file download, the method comprising:

acquiring a first file identifier of a file to be downloaded;

acquiring meta information of the file based on the first file identifier; the meta information of the file comprises a first file identifier of the file, at least one storage address and information of an arrangement sequence of at least one block file corresponding to the at least one storage address in the file before segmentation;

downloading the at least one block file from a storage server corresponding to the at least one storage address by using the at least one storage address; and

and restoring the at least one block file into a complete file and returning the complete file based on the information of the arrangement sequence of the at least one block file.

13. The method of claim 12, wherein the meta information of the file is stored in an information store; the obtaining the meta information of the file based on the first file identifier includes:

and searching and acquiring the meta information of the file in the information storage based on the first file identifier.

14. The method of claim 12, wherein when the meta information of the file is stored in a storage server; the obtaining the meta information of the file based on the first file identifier includes:

searching and acquiring a storage address of the meta information of the file in the information storage based on the first file identifier; and are

And downloading the meta information of the file from the storage server corresponding to the storage address of the meta information by using the storage address of the meta information.

15. The method of claim 12, wherein the meta information further comprises at least one second file identification corresponding to the at least one stored chunk file in association;

before the restoring the at least one block file to a complete file and returning the complete file based on the information of the arrangement order of the at least one block file, the method further includes:

calculating at least one third file identifier corresponding to the downloaded at least one block file;

when the third file identification of the block file is matched with the second file identification of the block file, determining that the block file passes verification;

and when the third file identifier of the block file is not matched with the second file identifier of the block file, determining that the block file does not pass the verification, re-downloading the block file, and replacing the re-downloaded block file with the block file which does not pass the verification.

16. The method of claim 12, wherein the meta information further includes information of at least one encoder corresponding to the at least one block file;

acquiring at least one decoder corresponding to the at least one encoder based on the information of the at least one encoder;

and using the at least one decoder to respectively restore the corresponding at least one block file into at least one block file before disguising.

17. The method of claim 12, wherein, when each of the at least one block file stored at the storage server masquerades as a block file having a target file format, respectively, the downloading of the at least one block file from the storage server corresponding to the at least one storage address using the at least one storage address comprises:

and downloading the partitioned files before disguising from the disguised partitioned files stored in the storage server corresponding to the storage address by using the position of each partitioned file before disguising in the corresponding disguised partitioned file.

18. The method of claim 12, wherein,

the number of the storage servers is multiple;

when each of the plurality of partitioned files is stored repeatedly in at least two of the plurality of storage servers;

the downloading, by using the at least one storage address, the at least one block file from the storage server corresponding to the at least one storage address includes:

downloading a corresponding block file from a storage server corresponding to a first storage address by using the first storage address in a plurality of storage addresses corresponding to each block file in a plurality of block files;

when downloading of the corresponding block file from the storage server corresponding to the first storage address by using the first storage address fails, downloading the corresponding block file from the storage server corresponding to the second storage address by using a second storage address of the plurality of storage addresses corresponding to the block file.

19. A file management device is in communication connection with a storage server and comprises a processor and an information memory;

the processor is configured to, when receiving a file upload request, perform:

acquiring a file to be uploaded;

dividing the file to be uploaded into at least one block file;

uploading the at least one block file to the storage server;

receiving at least one storage address corresponding to the at least one block file returned by the storage server;

storing meta information of the file;

when receiving a request for downloading a file, executing:

acquiring a first file identifier of a file to be downloaded;

acquiring meta information of the file based on the first file identifier;

downloading the at least one block file and the information of the arrangement sequence of the at least one block file from a storage server corresponding to the at least one storage address by using the at least one storage address, restoring the at least one block file into a complete file and returning the complete file;

the information memory is used for storing the meta information of the file or the storage address of the meta information; the meta information of the file comprises a first file identifier of the file, the at least one storage address and an arrangement sequence of the at least one block file corresponding to the at least one storage address in the file before the file is divided.