CN112583904B

CN112583904B - File uploading method, device, equipment and storage medium

Info

Publication number: CN112583904B
Application number: CN202011406833.XA
Authority: CN
Inventors: 李磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2020-12-04
Filing date: 2020-12-04
Publication date: 2023-05-19
Anticipated expiration: 2040-12-04
Also published as: CN112583904A

Abstract

The application discloses a file uploading method, device, equipment and storage medium, and relates to the technical field of cloud computing and cloud storage. The specific implementation scheme is as follows: receiving an uploading request corresponding to a target file from terminal equipment; determining a target block size corresponding to a target file according to a current network parameter, and sending the target block size to terminal equipment, wherein the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the target block size; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. According to the scheme, the file uploading success rate is improved.

Description

File uploading method, device, equipment and storage medium

Technical Field

The present disclosure relates to cloud computing and cloud storage technologies in the internet field, and in particular, to a method, an apparatus, a device, and a storage medium for uploading files.

Background

With development and popularization of cloud storage technology, more and more users like to store files in cloud servers. In practical application, when the file is larger, the uploading success rate of the file is lower.

In the prior art, in order to ensure successful uploading of a larger file, a cloud server provides a block uploading interface so that a terminal device divides the larger file into a plurality of data blocks and sends the data blocks to the cloud server one by one. In order to facilitate the subsequent file query process, the cloud server needs to generate and store file meta-information after receiving a plurality of data blocks, and identify the file meta-information through a plurality of indexes.

However, in the above manner, the index structure of the file meta-information cannot increase the number of blocks without limit, so that the cloud server limits the maximum number of blocks in the block uploading interface. Thus, when an oversized file is faced, even if the blocks are divided according to the maximum number of blocks, the size of each data block may still be large, and when the network environment is poor, the file upload success rate is still low.

Disclosure of Invention

The application provides a file uploading method, device and equipment and a storage medium.

In a first aspect, the present application provides a file uploading method, including:

receiving an uploading request corresponding to a target file from terminal equipment;

determining the size of a target block corresponding to the target file according to the current network parameters, and sending the size of the target block to the terminal equipment; the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the size of the target block;

And receiving the data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure.

In a second aspect, the present application provides a file uploading device, including:

the receiving and transmitting module is used for receiving an uploading request corresponding to the target file from the terminal equipment;

the determining module is used for determining the size of the target block corresponding to the target file according to the current network parameters;

the receiving and transmitting module is further configured to send the target block size to the terminal device; the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the size of the target block;

the transceiver module is further configured to receive the plurality of data blocks from the terminal device;

and the storage module is used for storing the meta information of the target file by adopting a single-layer index structure.

In a third aspect, the present application provides an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the first aspects.

In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of the first aspects.

In a fifth aspect, the present application provides a computer program product comprising: a computer program stored in a readable storage medium, from which it can be read by at least one processor of an electronic device, the at least one processor executing the computer program causing the electronic device to perform the method of the first aspect.

The file uploading method, device, equipment and storage medium provided by the application comprise the following steps: receiving an uploading request corresponding to a target file from terminal equipment; determining a target block size corresponding to a target file according to a current network parameter, and sending the target block size to terminal equipment, wherein the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the target block size; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. In the above process, the metadata of the target file is stored by adopting a single-layer index structure, so that the limitation on the maximum blocking number is eliminated, and the data volume uploaded each time can be reduced as much as possible. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, and the data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the file uploading success rate.

It should be understood that the description of this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.

Drawings

The drawings are for better understanding of the present solution and do not constitute a limitation of the present application. Wherein:

fig. 1 is a schematic diagram of a file uploading application scenario provided in the present application;

FIG. 2 is a schematic diagram of file partitioning;

FIG. 3 is a diagram illustrating an index structure of meta information of a file according to the prior art;

FIG. 4 is a schematic flow chart of a file uploading method provided in the present application;

FIG. 5 is a schematic diagram of an index structure of file meta-information provided in the present application;

FIG. 6 is a flowchart illustrating another method for uploading files provided in the present application;

fig. 7 is a schematic structural diagram of a file uploading device provided in the present application;

FIG. 8 is a schematic structural diagram of another document uploading device provided in the present application;

fig. 9 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The application provides a file uploading method, device, equipment and storage medium, which are applied to the technical fields of cloud storage, cloud computing and the like in the field of Internet so as to improve the success rate of file uploading.

Fig. 1 is a schematic diagram of a file uploading application scenario provided in the present application. As shown in fig. 1, the application scenario includes: terminal equipment and high in the clouds server. The terminal equipment is connected with the cloud server through a network.

The terminal device may be an electronic device with information processing capability, such as a smart phone, a tablet computer or a desktop computer. The terminal equipment can access the cloud server through a webpage or a client side, and send a file uploading request to the cloud server so as to achieve the purpose of uploading files to the cloud server. The terminal equipment can also access the cloud server through a webpage or a client side, and send a file inquiry request or a file downloading request to the cloud server so as to realize the purpose of inquiring/downloading files from the cloud server.

The cloud server of the present embodiment may be, for example, a network disk server that provides a network disk storage service to a user. The cloud server may employ a distributed storage technology or a server incorporating a blockchain. Of course, other storage techniques may be employed by the cloud server.

And after receiving the file uploading request sent by the terminal equipment, the cloud server receives the file to be uploaded and stores the file. After receiving a file downloading request sent by the terminal equipment, the cloud server inquires the requested file and issues the file to the terminal equipment. After receiving a file inquiry request sent by a terminal device, the cloud server inquires a file of the request and returns related information of the file to the terminal device.

The cloud server provides universal cloud storage service for users. "Universal" means that a user can store files of any type and size. Moreover, any file storage and access should provide the same service guarantee to the user, such as file upload success rate.

However, in practical application, the larger the file is, the lower the uploading success rate is. The main reason is that: when the file is bigger, the network interaction time between the terminal equipment and the cloud server is longer, so that the probability of data transmission errors on the network is also bigger. Even if the network protocol layer and the service layer have failed retransmission strategies, successful uploading cannot be guaranteed in percentage.

In order to ensure successful uploading of larger files, the cloud server provides a block uploading interface. The terminal equipment divides the larger file into a plurality of data blocks, and calls the uploading interface once for each data block so as to send the data block to the cloud server. The larger file is divided into a plurality of data blocks, so that the data volume transmitted between the terminal equipment and the cloud server in each interaction is smaller, and the uploading success rate of the file is improved to a certain extent.

In order to facilitate understanding of the technical solution of the present application, the file block uploading process is described in the following with reference to fig. 2 and 3.

The cloud server provides three uploading interfaces for the terminal equipment, namely an initial uploading interface, a block uploading interface and a submitting uploading interface.

When the terminal equipment receives a file uploading operation instruction input by a user, the terminal equipment sends a file uploading request to the cloud server by calling an initial uploading interface. And after receiving the file uploading request, the cloud server generates a globally unique index identifier for the file. In one aspect, the cloud server notifies the terminal device of the index identifier of the file, so that the terminal device can associate the index identifier of the file to the block transmission process of the file. On the other hand, the cloud server can store the index identifier of the file into the meta information of the file, so that the file can be queried through the index identifier of the file in the follow-up file querying, downloading and other processes.

After the terminal device obtains the index identifier of the file from the cloud server, the file to be uploaded can be divided into a plurality of data blocks. FIG. 2 is a schematic diagram of file partitioning. As shown in fig. 2, for example, it is assumed that a file is divided into data block 0, data block 1, data blocks 2, … …, and data block n. And the terminal equipment calls a block uploading interface once for each data block so as to upload the data block to the cloud server.

Correspondingly, after receiving each data block, the cloud server stores the data block in a slicing way. Specifically, the cloud server randomly allocates an index identifier to each data block, divides each data block into a plurality of data slices according to the slice size required by the storage system (as shown in fig. 2, divides the data block 0 into a data slice 0, a data slice 1, a data slice 2, … … and a data slice m), and also randomly allocates an index identifier to each data slice. And after the cloud server stores each data block, the meta information of the data block is stored. The meta information of the data block includes an index identifier of the data block, fingerprint information, an index identifier of each data piece in the data block, and the like.

After the terminal equipment finishes uploading all the data blocks, the terminal equipment informs the cloud server of finishing uploading all the data blocks of the file by calling the submitting uploading interface. And the cloud server organizes the meta information of all the data blocks corresponding to the file into the meta information of the file, and stores the meta information of the file, so that the uploading process of the whole file is completed.

Fig. 3 is a schematic diagram of an index structure of meta information of a file in the prior art. As shown in fig. 3, the meta information of the file stored in the cloud server includes: index identification of the file, size of the file, fingerprint information of the file, etc., and index identification list of all data blocks is also stored in meta information of the file. Thus, when the user subsequently inquires the file, the cloud server can acquire the index identification of each data block corresponding to the file according to the meta-information of the stored file. The meta information of each data block can be further obtained according to the index identifier of each data block, and the index identifiers of the data pieces corresponding to the data block are obtained from the meta information of each data block, so that the content of the file is queried.

As can be seen from the above procedure, in order to facilitate the file query/download process, the cloud server needs to use at least two layers of index structures (such as the index structures shown in fig. 3) to store the file information.

Because the cloud server needs to store index identifiers of all data blocks in meta-information of the file, the storage mode causes that the number of blocks of the file cannot be increased without limit. Therefore, the maximum number of tiles is typically limited in the tile upload interface provided by the cloud server. For example, some cloud servers limit the maximum number of blocks to 10000, and some cloud servers limit the maximum number of blocks to 2000. This limitation of the number of tiles limits the minimum value of the tile size. For example, if the size of the file to be uploaded is 10GB, the size of each data block is 5MB (here, 1 gb=1000 MB) if the blocks are divided by 2000 pieces of the maximum number of blocks. If the size of the file to be uploaded is 100GB, and the maximum number of blocks is 2000, the size of each data block is 50MB. When the network environment is poor, the success rate of transmitting 50MB data blocks in a single request is also not guaranteed. According to online actual data statistics, when the file is larger than 10GB, the file uploading failure rate can reach more than 10%, and the user experience is seriously affected.

It can be seen that in the above prior art, when faced with oversized files, the size of each data block may still be large (e.g., greater than 10MB, even greater than 50 MB) even if divided by the maximum number of chunks. Under the condition of poor network environment, the uploading success rate of the files is still low even if the above-mentioned block transmission process is adopted.

In addition, in the above prior art, since a large number of meta information of data blocks need to be stored in the meta information of the file, the capacity of the meta information of the file is increased, on one hand, a large storage space needs to be occupied, and on the other hand, when the file needs to be queried, the calculation amount needed to analyze the meta information of the file is also large, which results in lower file query efficiency.

In order to solve at least one of the above technical problems, the present application provides a file uploading method, whose core idea is to reduce the data amount of each uploading as much as possible, that is, to divide the data blocks of the file as small as possible. For the same file, the smaller the divided data blocks, the greater the number of corresponding data blocks. Therefore, the core problem to be solved in the present application is how to break the limitation of the maximum number of blocks in the prior art. The file uploading method can cancel the limitation of the maximum blocking number by improving the index structure of the meta-information of the file, so that the data volume uploaded each time can be reduced as much as possible. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, and the data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the file uploading success rate.

The technical solutions of the present application are described in detail below with reference to several specific embodiments. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.

Fig. 4 is a flow chart of a file uploading method provided in the present application. The method of the present embodiment may be performed by a cloud server. As shown in fig. 4, the method of the present embodiment may include:

s401: and receiving an uploading request corresponding to the target file from the terminal equipment.

Specifically, when receiving a file uploading instruction input by a user, the terminal equipment determines a target file to be uploaded and sends an uploading request to a cloud server to request the uploading of the target file. The target file to be uploaded includes, but is not limited to: text files, audio files, video files, etc.

The terminal device may send an upload request corresponding to the target file by invoking an initial upload interface provided by the cloud server. And the cloud server receives an uploading request corresponding to the target file.

Optionally, after receiving the upload request, the cloud server may generate a globally unique index identifier for the target file. The cloud server can also inform the terminal device of the index identification of the target file, so that the terminal device can associate the index identification of the target file to the block transmission process of the file. For example, when uploading each data block, the terminal device carries an index identifier of the target file to indicate that the currently transmitted data block is a data block in the target file.

The index identifier of the target file may also be used to store meta-information of the target file, so that the target file may be queried through the index identifier in a subsequent file querying, downloading, etc. process.

S402: and determining the size of the target block corresponding to the target file according to the current network parameter, and sending the size of the target block to the terminal equipment, wherein the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameter, and the size of each data block is smaller than or equal to the size of the target block.

The difference between the present embodiment and the prior art is that, in the present embodiment, when determining the target block size corresponding to the target file, the limitation of the maximum block number is not considered, but the smaller and better each data block is divided as much as possible, so as to ensure successful uploading of each data block. Illustratively, the cloud server may configure a default smaller chunk size (e.g., 1 MB) to ensure that the data chunks may also be successfully uploaded in a weak network environment.

Of course, the smaller the block, the more the IO number, and the throughput will be reduced accordingly. Therefore, in the embodiment of the present application, in order to balance the success rate and the throughput rate, the cloud server may determine the target block size according to the current network parameter. That is, the cloud server determines the size of the target block that can be successfully uploaded in the current network environment.

After receiving the uploading request corresponding to the target file, the cloud server can acquire the current network parameters according to the uploading request. For example, when the terminal device sends the upload request, the current network parameters may be carried in the upload request. Or the cloud server measures and obtains the current network parameters according to the receiving condition of the uploading request. Or after receiving the uploading request, the cloud server acquires the current network parameters through a message interaction process with the terminal equipment. This embodiment is not limited thereto.

In this embodiment, the network parameter may be any parameter for indicating the quality of the network environment. Optionally, the network parameters may include one or more of the following: location information of the terminal device, operator information accessed by the terminal device, network transmission rate, network signal strength, and the like.

When the network environment quality indicated by the current network parameters is good, the target block size can be set to be a larger value; when the network environment quality indicated by the current network parameters is poor, the target chunk size may be set to a smaller value. The size of the target block is dynamically determined according to the current network parameters, so that higher throughput rate can be ensured and uploading efficiency is improved on the premise of ensuring successful uploading of the file.

In one possible implementation manner, the block size threshold corresponding to different network parameters may be determined according to the historical file uploading condition or according to the file uploading test condition. The block size threshold corresponding to each network parameter indicates that: in the network environment corresponding to the network parameters, the maximum block size of successful uploading can be ensured. For example, if the block size threshold corresponding to a certain network parameter is 10MB, it indicates that the data block of maximum 10MB can be successfully uploaded in the network environment corresponding to the network parameter, and if the data block is greater than 10MB, the successful uploading cannot be ensured.

When determining the size of the target block corresponding to the target file according to the current network parameter, the cloud server can take the block size threshold corresponding to the current network parameter as the size of the target block. Alternatively, the target chunk size is set to some value less than the chunk size threshold. For example, if the block size threshold corresponding to the current network parameter is 10MB, the cloud server may determine the target block size to be 10MB, or 8MB, or 5MB, or the like.

And after determining the target block size corresponding to the target file, the cloud server sends the target block size to the terminal equipment so that the terminal equipment divides the target file into a plurality of data blocks according to the target block size. Wherein the size of each data block is less than or equal to the target block size.

In this embodiment, the size of the target block corresponding to the target file is determined according to the current network parameter, so that the determined size of the target block meets the current network environment, and it is ensured that the data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the success rate of file uploading.

S403: and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure.

After the terminal equipment divides the target file into a plurality of data blocks, each data block can be uploaded to the cloud server through the block uploading interface. And after receiving each data block, the cloud server stores each data block in a slicing way. And after the cloud server receives the plurality of data blocks, generating and storing meta-information of the target file.

The present embodiment is different from the prior art in that the present embodiment stores the meta information of the target file not using the index structure of at least two layers in the prior art any more, but using the index structure of a single layer. It should be understood that "single-layer index structure" herein refers to that only file-level information is stored in meta-information of a target file, and index identifications of respective data blocks are not stored. That is, the meta information of each data block is not stored in the meta information of the target file in the present embodiment.

Fig. 5 is a schematic diagram of an index structure of file meta information provided in the present application. As shown in fig. 5, only file-level information is stored in meta information of the target file, for example: the size of the file, the index identification of the file, fingerprint information of the file, etc. The difference from the file meta-information shown in fig. 3 is that the index identification of each data block is not stored.

In this embodiment, "the index identifier of each data block is not stored" means that the index identifier of each data block is not explicitly stored. Of course, in order to index information of each data block of the target file in the subsequent file query flow, the index identifier of each data block may be implicitly stored in meta information of the target file, or the index identifier of each data block may be implicitly indicated. The present embodiment is not limited in terms of implicit storage or implicit indication, and one possible implementation may be found in the detailed description of the embodiments that follow.

Because the index identification list of the data blocks is not stored in the meta information of the target file in the embodiment, the meta information of the target file is not affected by the number of the blocks, and therefore the maximum number of the blocks is not limited in the embodiment. That is, the present embodiment stores meta information of a target file by using a single-layer index structure to cancel the restriction on the maximum number of blocks, thereby enabling "reducing the amount of data per upload as much as possible" to be realized.

The file uploading method provided by the embodiment comprises the following steps: receiving an uploading request corresponding to a target file from terminal equipment; determining a target block size corresponding to a target file according to a current network parameter, and sending the target block size to terminal equipment, wherein the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the target block size; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. In the above process, the metadata of the target file is stored by adopting a single-layer index structure, so that the limitation on the maximum blocking number is eliminated, and the data volume uploaded each time can be reduced as much as possible. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, and the data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the file uploading success rate.

Based on any of the above embodiments, the technical solution of the present application will be described in more detail below with reference to fig. 6.

Fig. 6 is a flow chart of another file uploading method provided in the present application. As shown in fig. 6, the method of the present embodiment includes:

s601: and the terminal equipment sends an uploading request corresponding to the target file to the cloud server.

Correspondingly, the cloud server receives an uploading request corresponding to the target file from the terminal equipment.

S602: and the cloud server determines the size of the target block corresponding to the target file according to the current network parameters. The target block size is smaller than or equal to a block size threshold corresponding to the current network parameter.

Optionally, after receiving the upload request corresponding to the target file, the cloud server may further determine whether the upload request is a first upload request for the target file. That is, it is determined whether the target file is uploaded for the first time or re-uploaded.

In some scenarios, if it is determined that the upload request is the first upload request for the target file, S602 is performed. That is, if the target file is determined to be uploaded for the first time, the current network parameter is obtained, and the target block size corresponding to the target file is determined according to the current network parameter.

In other scenarios, if it is determined that the upload request is not the first upload request for the target file, then the target file is indicated as being re-uploaded. In this case, the historical block size corresponding to the target file may be obtained, and the historical block size may be adjusted to obtain the target block size corresponding to the target file. For example, the historical chunk size may be adjusted to a smaller value, with the adjusted chunk size being the target chunk size.

In this embodiment, if the target file fails to be uploaded due to the larger target block size when the target file is uploaded for the first time, when the terminal device initiates the re-uploading, the cloud server may issue the smaller target block size to the terminal device (for example, the target block size during the re-uploading may be half of the block size corresponding to the first uploading), so as to ensure that the file is successfully uploaded.

S603: and the cloud server sends the target block size to the terminal equipment.

S604: the terminal equipment divides the target file into a plurality of data blocks according to the target block size, wherein the size of each data block is smaller than or equal to the target block size.

S605: and the terminal equipment respectively sends each data block to the cloud server.

Correspondingly, the cloud server receives each data block from the terminal device.

S606: and the cloud server stores each data block according to the index identification of the target file, and generates and caches meta-information of each data block.

For any first data block in the plurality of data blocks, the cloud server may store the first data block in the following feasible manner, and generate and cache meta information of the first data block.

(1) And determining the index identification of the first data block according to the index identification of the target file.

Optionally, when the terminal device sends each data block to the cloud server, the terminal device carries the serial number of the data block. The cloud server can calculate the index identifier of the target file and the serial number of each data block by adopting first hash operation, and determines the index identifier of the data block. The following formula is shown:

PartID _i ＝Func1(FileID,i)

where i is the sequence number of the data block, partID _i For index identification of the i-th data block, fileID is index identification of the target file, func1 () is the first hash operation.

(2) Dividing the first data block into a plurality of data slices, determining the index identification of each data slice according to the index identification of the first data block, and storing the plurality of data slices according to the index identifications of the plurality of data slices.

Optionally, the cloud server may divide the first data block into a plurality of data slices according to a slice size required by the storage system of the cloud server. And the cloud server calculates the index identifier of the first data block and the serial number of each data piece by adopting a second hash operation, and determines the index identifier of each data piece. The following formula is shown:

SliceID _j ＝Func2(PartID _i ,j)

wherein j is the serial number of the data sheet, and SliceID _j Index identification for the jth slice in the ith block of data, partID _i For index identification of the ith data block, func2 () is the second hash operation.

(3) And generating and caching the meta information of the first data block, wherein the meta information of the first data block comprises index identification and fingerprint information of the first data block.

The present embodiment is further different from the prior art in that, after the cloud server generates the meta information of the data block, the meta information of the data block is not stored in a persistent manner (i.e., the meta information of the data block is not stored in the meta information of the target file), but the meta information of the data block is cached in the cache system. For example, the cache system may be a redis cache system. The meta-information of the cached data block may be used to verify whether the target file was successfully uploaded.

S607: and the terminal equipment sends a file submitting request to the cloud server.

Correspondingly, the cloud server receives a file submitting request from the terminal equipment. The file submitting request is used for informing the cloud server that all data blocks of the target file are uploaded. Optionally, the terminal device may send a file submission request to the cloud server by calling the submission upload interface.

S608: and when the cloud server verifies that the data blocks are all uploaded correctly according to the meta-information of the data blocks, generating and storing the meta-information of the target file.

And the cloud server checks whether all the data blocks of the target file are uploaded correctly according to the meta information of the data blocks cached in the cache system. If all the data blocks of the target file are determined to be uploaded correctly, generating and storing meta-information of the target file. The meta information of the target file adopts a single-layer index structure, the meta information of the target file includes the index identifier of the target file, and does not include the meta information of each data block in the target file (i.e., does not include the index identifier of each data block in the target file).

S609: and the cloud server clears the meta-information of the cached data blocks.

In this embodiment, the index identifier of each data block is determined according to the index identifier of the target file and the first hash operation, and the index identifier of each data slice is determined according to the index identifier of the data block and the second hash operation, so that the index identifiers of each data block and each data slice corresponding to the target file can be obtained by calculation only according to the index identifier of the target file, that is, the content of the target file can be obtained by indexing only according to the index identifier of the target file. Therefore, the metadata of the target file is stored by adopting a single-layer index structure, so that the limit on the maximum blocking number is eliminated, and the data volume uploaded each time can be reduced as much as possible. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, and the data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the file uploading success rate.

The following describes the query flow of the target file based on the embodiment shown in fig. 6.

When receiving a file query operation input by a user, the terminal equipment determines a target file to be queried and sends a file query request corresponding to the target file to the cloud server. The file query request is for requesting the contents of a query target file. And the cloud server acquires the index identification of the target file according to the meta information of the target file. And the cloud server acquires index identifiers of a plurality of data blocks corresponding to the target file according to the index identifiers of the target file and the first hash operation. Further, the cloud server obtains index identifiers of a plurality of data pieces corresponding to each data block according to the index identifiers of each data block and the second hash operation. Furthermore, the cloud server may obtain the content of the target file from the storage space corresponding to the index identifier of each data piece. Through the process, the query flow of the target file is realized.

In this embodiment, since the meta information of each data block is not stored in the meta information of the target file, the storage amount of the meta information of the file is reduced, the calculation amount for analyzing the meta information of the file during the file query is reduced, and the file query efficiency is improved.

In practical application, a phenomenon that a certain data block is re-uploaded may exist in the file uploading process. For example, a file may have been partially changed during the upload process, and the changed data block may be re-uploaded. On the basis of the embodiment shown in fig. 6, since the index identifier of the data block is obtained by calculating the index identifier of the target file and the sequence number of the data block by using the first hash operation. Thus, when a certain data block is re-uploaded, the index identity of the re-uploaded data block and the index identity of the previously uploaded data block are the same (because the calculation formula of the index identity of the data block is the same).

If the cloud server adopts a distributed storage system, since the distributed storage system cannot guarantee the writing time sequence of data, when a certain data is repeatedly received, an overwriting mode is generally adopted. Thus, when the data block n is re-uploaded, it may occur that a new data block n is written first, and an old data block n is written after and overwrites the new data block n, resulting in file data confusion.

In this embodiment, in order to avoid the problem of data confusion caused by data block retransmission, the cloud server may adopt a locking mode when receiving each data block and storing the data block.

In one possible implementation manner, after the cloud server receives the first data block and determines the index identifier of the first data block, it may also determine whether the first data block is received for the first time according to the index identifier of the first data block. For example, it may be queried whether meta-information of the first data block exists in the cache system, and if so, the description is not first received, and if not, the description is first received.

If it is determined that the first data block is received for the first time, the first data block may be stored according to the procedure of the above embodiment.

If it is determined that the first data block is not received for the first time, the content of the first data block is compared with the content of the second data block. Wherein the second data block is the first received data block having the same index identification as the first data block. And if the comparison result is consistent, the first data block is considered to be successfully uploaded. If the comparison result is inconsistent, the first data block is considered to be failed to upload, and an upload failure message is sent to the terminal equipment.

Optionally, after the cloud server sends the upload failure message to the terminal device, the cloud server may also send a prompt message to the terminal device, where the prompt message is used to prompt the user to interrupt the transmission of the target file and re-upload the target file.

In this embodiment, by performing the locking operation when writing the data block, the problem of data confusion caused by the retransmission of the data block can be avoided, thereby ensuring the correct uploading of the file.

Fig. 7 is a schematic structural diagram of a file uploading device provided in the present application. The file uploading device of the present embodiment may be in the form of software and/or hardware. The file uploading device of the embodiment can be arranged in the cloud server. As shown in fig. 7, the file uploading device 10 of the present embodiment may include: a transceiver module 11, a determination module 12 and a storage module 13.

The transceiver module 11 is configured to receive an upload request corresponding to a target file from a terminal device;

the determining module 12 is configured to determine, according to a current network parameter, a target block size corresponding to the target file;

the transceiver module 11 is further configured to send the target block size to the terminal device; the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the size of the target block;

The transceiver module 11 is further configured to receive the plurality of data blocks from the terminal device;

the storage module 13 is configured to store meta information of the target file by using a single-layer index structure.

In a possible implementation manner, the transceiver module 11 is specifically configured to: receiving each of the plurality of data blocks from the terminal device separately;

the storage module 13 is specifically configured to:

storing each data block according to the index identification of the target file, and generating and caching meta-information of each data block;

generating and storing the meta information of the target file when the plurality of data blocks are checked to be uploaded correctly according to the meta information of the plurality of data blocks, and clearing the cached meta information of the plurality of data blocks;

the meta information of the target file comprises an index identifier of the target file.

In a possible implementation manner, the storage module 13 is specifically configured to: for any first data block in the plurality of data blocks, determining an index identifier of the first data block according to the index identifier of the target file;

dividing the first data block into a plurality of data pieces, determining an index identifier of each data piece according to the index identifier of the first data block, and storing the plurality of data pieces according to the index identifiers of the plurality of data pieces;

And generating and caching the meta information of the first data block, wherein the meta information of the first data block comprises index identification and fingerprint information of the first data block.

In a possible implementation manner, the storage module 13 is specifically configured to:

calculating the index identifier of the target file and the serial number of the first data block by adopting a first hash operation, and determining the index identifier of the first data block;

and calculating the index identifier of the first data block and the serial number of each data slice by adopting a second hash operation, and determining the index identifier of each data slice.

Fig. 8 is a schematic structural diagram of another file uploading device provided in the present application. As shown in fig. 8, on the basis of the embodiment shown in fig. 7, the apparatus of this embodiment further includes: the module 14 is acquired.

The transceiver module 11 is further configured to: receiving a file inquiry request from the terminal equipment, wherein the file inquiry request is used for requesting to inquire the content of the target file;

the acquisition module 14 is configured to: acquiring an index identifier of the target file according to the meta information of the target file;

according to the index identification of the target file and the first hash operation, obtaining index identifications of a plurality of data blocks corresponding to the target file, and according to the index identification of each data block and the second hash operation, obtaining index identifications of a plurality of data pieces corresponding to each data block;

And acquiring the content of the target file from the storage space corresponding to the index identifiers of the data pieces.

In a possible implementation manner, the determining module 12 is specifically configured to:

determining whether the upload request is a first upload request for the target file;

if the uploading request is determined to be the first uploading request aiming at the target file, acquiring a current network parameter, and determining a target block size corresponding to the target file according to the current network parameter.

In a possible implementation, the determining module 12 is further configured to:

if the uploading request is not the first uploading request aiming at the target file, acquiring the historical block size corresponding to the target file, and adjusting the historical block size to obtain the target block size corresponding to the target file.

In a possible implementation, the storage module 13 is further configured to:

determining whether the first data block is received for the first time according to the index identification of the first data block;

if the first data block is determined not to be received for the first time, comparing the content of the first data block with the content of a second data block, wherein the second data block is the data block which is received for the first time and has the same index identification as the first data block;

The transceiver module 11 is further configured to: and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.

In a possible implementation manner, the transceiver module 11 is further configured to: and sending prompt information to the terminal equipment, wherein the prompt information is used for prompting a user to interrupt the transmission of the target file and re-upload the target file.

In a possible implementation, the current network parameters include at least one of the following: the location information of the terminal equipment, the information of an operator to which the terminal equipment is connected, the network transmission rate and the network signal strength.

According to embodiments of the present application, an electronic device and a readable storage medium are also provided. The electronic device may act as a cloud server.

As shown in fig. 9, a block diagram of an electronic device according to a file uploading method according to an embodiment of the present application is shown. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.

As shown in fig. 9, the electronic device includes: one or more processors 101, memory 102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). In fig. 9, a processor 101 is taken as an example.

Memory 102 is a non-transitory computer-readable storage medium provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the file uploading method provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the file upload method provided by the present application.

The memory 102 is used as a non-transitory computer readable storage medium, and may be used to store a non-transitory software program, a non-transitory computer executable program, and modules, such as program instructions/modules (e.g., the transceiver module 11, the determining module 12, and the storage module 13 shown in fig. 7, and the acquiring module 14 shown in fig. 8) corresponding to the file uploading method in the embodiments of the present application. The processor 101 executes various functional applications of the server and data processing, i.e., implements the file upload method in the above-described method embodiment, by running non-transitory software programs, instructions, and modules stored in the memory 102.

The memory 102 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the electronic device of the file upload method, and the like. In addition, the memory 102 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 102 may optionally include memory located remotely from processor 101, which may be connected to the electronic device of the file upload method via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the file uploading method may further include: an input device 103 and an output device 104. The processor 101, memory 102, input device 103, and output device 104 may be connected by a bus or otherwise, for example in fig. 9.

The input device 103 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device of the file upload method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. input devices. The output device 104 may include a display apparatus, auxiliary lighting devices (e.g., LEDs), haptic feedback devices (e.g., vibration motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions disclosed in the present application can be achieved, and are not limited herein.

The present application also provides a computer program product comprising: a computer program stored in a readable storage medium, from which at least one processor of an electronic device can read, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any one of the embodiments described above.

The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. The file uploading method is applied to a cloud server and is characterized by comprising the following steps:

Each data block in the plurality of data blocks is received from the terminal equipment respectively, and is stored according to the index identification of the target file, so that meta information of each data block is generated and cached; the meta information of each data block comprises an index identification of the data block;

2. The method of claim 1, wherein for any first data block of the plurality of data blocks, storing the first data block according to the index identifier of the target file, generating and caching meta information of the first data block, includes:

determining the index identification of the first data block according to the index identification of the target file;

3. The method of claim 2, wherein determining the index identification of the first data block from the index identification of the target file comprises:

determining the index identifier of each data slice according to the index identifier of the first data block, including:

4. The method of claim 3, further comprising, after storing the meta information of the target file using a single-layer index structure:

receiving a file inquiry request from the terminal equipment, wherein the file inquiry request is used for requesting to inquire the content of the target file;

acquiring an index identifier of the target file according to the meta information of the target file;

5. The method according to any one of claims 1 to 4, wherein determining, according to current network parameters, a target block size corresponding to the target file includes:

6. The method of claim 5, the method further comprising:

7. The method according to any one of claims 2 to 4, further comprising, after determining the index identifier of the first data block from the index identifier of the target file:

and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.

8. The method of claim 7, further comprising, after sending an upload failure message to the terminal apparatus:

and sending prompt information to the terminal equipment, wherein the prompt information is used for prompting a user to interrupt the transmission of the target file and re-upload the target file.

9. The method of any of claims 1-4, 6, 8, the current network parameters comprising at least one of: the location information of the terminal equipment, the information of an operator to which the terminal equipment is connected, the network transmission rate and the network signal strength.

10. The utility model provides a file uploading device, sets up in high in the clouds server, its characterized in that includes: the device comprises a receiving and transmitting module, a determining module and a storage module; wherein,,

the storage module is used for storing the meta information of the target file by adopting a single-layer index structure;

the transceiver module is specifically configured to: receiving each of the plurality of data blocks from the terminal device separately;

the storage module is specifically used for:

storing each data block according to the index identification of the target file, and generating and caching meta-information of each data block; the meta information of each data block comprises an index identification of the data block;

11. The apparatus of claim 10, wherein the storage module is specifically configured to: for any first data block in the plurality of data blocks, determining an index identifier of the first data block according to the index identifier of the target file;

12. The apparatus of claim 11, wherein the storage module is specifically configured to:

13. The apparatus of claim 12, the apparatus further comprising: an acquisition module;

the transceiver module is further configured to: receiving a file inquiry request from the terminal equipment, wherein the file inquiry request is used for requesting to inquire the content of the target file;

the acquisition module is used for: acquiring an index identifier of the target file according to the meta information of the target file;

14. The apparatus according to any one of claims 10 to 13, wherein the determining module is specifically configured to:

15. The apparatus of claim 14, the determination module further to:

16. The apparatus of any of claims 11 to 13, the storage module further to:

the transceiver module is further configured to: and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

18. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 9.