CN116455889A

CN116455889A - File uploading processing method, device and system, electronic equipment and storage medium

Info

Publication number: CN116455889A
Application number: CN202310272756.0A
Authority: CN
Inventors: 李磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2023-03-17
Filing date: 2023-03-17
Publication date: 2023-07-18

Abstract

The disclosure provides a file uploading processing method, device, system, electronic equipment and storage medium, and relates to the technical fields of file transmission, file storage and the like. The specific implementation scheme is as follows: in the process of receiving each data block in a plurality of data blocks of the target file uploaded by the client, if the data reading failure of the current data block is detected and determined; determining the breakpoint position of the current data block which fails to be read; and acquiring data transmitted subsequently to the breakpoint position of the current data block based on the breakpoint position of the current data block. The technology disclosed by the invention can effectively improve the success rate of file uploading.

Description

File uploading processing method, device and system, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of computers, in particular to the technical fields of file transmission, file storage and the like, and particularly relates to a file uploading processing method, device, system, electronic equipment and storage medium.

Background

Because the network disk product has the advantages of large storage capacity and the like, the network disk product can help users store a great amount of data, and is favored by various users.

In recent years, the demands for uploading user files have also changed in network disk usage, including restrictions on individual file sizes, which have been increasing, for example, a size restriction of 100GB, 1TB, 2TB has been opened successively, and a restriction of 4TB, 8TB, etc. has been likely to be opened again in the expected time of visibility. However, as the file size limit increases, the user's view increases the rate of file upload failures. Because the larger the single file, the higher the requirements for network stability, client hardware stability, client and server business service stability during the upload process. In order to improve the success rate of uploading, a large file can be split into a plurality of data blocks for uploading. And in order to further increase the upload success rate. In the upload logic of each file, a failed retry strategy is set.

Disclosure of Invention

The disclosure provides a file uploading processing method, device, system, electronic equipment and storage medium.

According to an aspect of the present disclosure, there is provided a method for processing uploading of a file, including:

in the process of receiving each data block in a plurality of data blocks of the target file uploaded by the client, if the data reading failure of the current data block is detected and determined;

Determining the breakpoint position of the current data block which fails to be read;

and acquiring data transmitted subsequently to the breakpoint position of the current data block based on the breakpoint position of the current data block.

According to another aspect of the present disclosure, there is provided a method for processing uploading of a file, including:

in the process of uploading each data block in a plurality of data blocks of a target file to a server, detecting and determining that the uploading of the current data block fails;

acquiring the breakpoint position of the current data block corresponding to the uploading failure;

and continuously transmitting the data of the current data block based on the breakpoint position of the current data block.

According to still another aspect of the present disclosure, there is provided a server including:

the detection module is used for detecting and determining that the data of the current data block fails to be read in the process of receiving each data block in a plurality of data blocks of the target file uploaded by the client;

a determining module, configured to determine a breakpoint position of the current data block that fails to be read;

and the acquisition module is used for acquiring data transmitted subsequently to the breakpoint position of the current data block based on the breakpoint position of the current data block.

According to yet another aspect of the present disclosure, there is provided a client device including:

The detection module is used for detecting and determining that the uploading of the current data block fails in the process of uploading each data block in the plurality of data blocks of the target file to the server;

the acquisition module is used for acquiring the breakpoint position of the current data block corresponding to the uploading failure;

and the transmission module is used for continuously transmitting the data of the current data block based on the breakpoint position of the current data block.

According to still another aspect of the present disclosure, there is provided a file upload processing system, including a client device and a server, where the client device and the server are communicatively connected to implement uploading each of a plurality of data blocks of a target file to the server; the server employs the server as described in the above aspect, and the client device employs the client device as described in the above aspect.

According to still another aspect of the present disclosure, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the aspects and methods of any one of the possible implementations described above.

According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of the aspects and any possible implementation described above.

According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the aspects and any one of the possible implementations described above.

According to the technology disclosed by the invention, the success rate of file uploading can be effectively improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a relationship between a file, a data block, and a preset fragment provided by the present disclosure;

FIG. 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

FIG. 8 is a schematic diagram according to a seventh embodiment of the present disclosure;

FIG. 9 is a schematic diagram according to an eighth embodiment of the present disclosure;

FIG. 10 is a schematic diagram according to a ninth embodiment of the present disclosure;

FIG. 11 is a schematic diagram according to a tenth embodiment of the present disclosure;

fig. 12 is a block diagram of an electronic device used to implement the methods of embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It will be apparent that the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments in this disclosure without inventive faculty, are intended to be within the scope of this disclosure.

It should be noted that, the terminal device in the embodiments of the present disclosure may include, but is not limited to, smart devices such as a mobile phone, a personal digital assistant (Personal Digital Assistant, PDA), a wireless handheld device, and a Tablet Computer (Tablet Computer); the display device may include, but is not limited to, a personal computer, a television, or the like having a display function.

In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure; as shown in fig. 1, the present embodiment provides a method for uploading a file, which may be applied to a server side to implement uploading processing of a large file, and specifically may include the following steps:

s101, detecting and determining that the data of the current data block fails to be read in the process of receiving each data block of a plurality of data blocks of the target file uploaded by the client;

s102, determining the breakpoint position of the current data block which is failed to be read;

S103, based on the breakpoint position of the current data block, acquiring data transmitted subsequently to the breakpoint position of the current data block.

The file uploading processing method is suitable for uploading large files. The large file in this embodiment may refer to that the size, that is, the storage capacity, of the file is greater than a preset capacity threshold, which may be set according to a specific application scenario and a specific requirement, for example, may be 500M or 1T, or other values.

Because the large file is directly uploaded, the failure rate is high due to the image of the network environment and the like. Therefore, the client side can divide the large file (file) into a plurality of data blocks (parts) by presetting a strategy, and the data blocks are uploaded at granularity. The upload of each data block may be provided with an upload retry strategy. I.e. if a certain data block fails to be uploaded, the uploading can be performed again. For example, a preset policy may require that a large file be evenly split into multiple data blocks. Alternatively, the set capacity threshold may be used as a splitting unit to split the large file into a plurality of data blocks in order from front to rear, and the size of the remaining last data block may be smaller than the set capacity threshold. Of course, alternatively, according to the actual scene, the large file may be unevenly segmented to obtain a plurality of different data blocks. The information of the data block can be recorded no matter which strategy is adopted and stored in the meta information table of the file, so that the data block is convenient to upload. For example, in the meta information table of the file, a part list may be provided in which information such as the order identification number of each data block part, the size of the part, and the like is included.

But as the demand of users continues to change, more and more files with larger storage capacity appear, which may be referred to as oversized files, for example. In addition, for convenience of management, the number of the segmented data blocks cannot be excessive, and after the oversized file is split into a plurality of data blocks, the size of each data block is still very large. According to online statistics, even if a failed retransmission strategy is set, the uploading success rate of the oversized file still cannot reach 90%. Based on the method, the processing method on the file is provided, so that the uploading success rate of the file is improved.

For any target file no matter how large the storage capacity is, in this embodiment, each file is still split into multiple data blocks for transmission at the client side. Specifically, on the server side, in the process of receiving each data block of a plurality of data blocks of the target file uploaded by the client, if the data reading failure of the current data block is detected and determined, the breakpoint position of the current data block with the reading failure needs to be determined, and then the data transmitted subsequently to the breakpoint position of the current data block is acquired based on the breakpoint position of the current data block. That is, in the technical solution of this embodiment, in the uploading of each data block of the target file, breakpoint continuous transmission can be supported, so that the uploading success rate of the data block with a larger storage capacity can be effectively improved, and the uploading success rate of the file can be effectively improved.

By adopting the technical scheme, the file uploading processing method can support breakpoint continuous transmission when uploading fails in any data block uploading process in the file, avoid retransmission of the data block, effectively improve the uploading success rate of the data block, and further effectively improve the uploading success rate of the file.

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure; the present embodiment provides a method for uploading files, and further describes the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiment shown in fig. 1. As shown in fig. 2, the method for processing file uploading in this embodiment may specifically include the following steps:

s201, detecting whether reading is normal or not in the process of receiving each data block of a plurality of data blocks of a client uploading target file; if yes, go to step S202; if the reading fails, step S203 is executed;

s202, storing the read data of the current data block to a storage system by taking the preset storage capacity of the preset fragments as storage granularity; and returning to the step S201 to continue detecting the subsequent reading process until the data storage of the current data block is finished.

S203, storing the data of which the data quantity does not reach the preset storage capacity of the preset fragments in the cache, wherein the data of the read current data block; step S204 is executed;

when the target file is larger, a plurality of data blocks of the target file are also larger, so that in the uploading process of each data block of the target file, the client side adopts a streaming reading mode to read and upload data in the data blocks of the target file to the server side. Correspondingly, on the server side, the server also adopts a streaming reading mode, and stores data in a data block of the target file while reading.

Specifically, during the reading process of each data block, the server may detect whether the storage capacity of the data of the current data block that has been received reaches the total storage capacity of the current data block that is acquired in advance, so as to determine whether the uploading of the data of the current data block is completed. If the storage capacity of the received data of the current data block reaches the total storage capacity of the current data block, determining that the uploading of the current data block is finished; otherwise, determining that the current data block is not uploaded. Then, whether the data of the current data block uploaded by the client can be read or not is detected, and if the data can be read, the reading is normal. If the data of the current data block is not completely uploaded, but the data of the current data block cannot be read, indicating that the reading is failed.

In this embodiment, when the reading is successful, in the process of streaming reading, it may be detected whether the storage capacity of the data of the current data block read reaches the preset storage capacity of the preset fragment; and if the data of the current data block is obtained, taking the preset storage capacity of the preset fragments as the storage granularity, and storing the read data of the current data block into a storage system. In a similar manner, all data of the current data block can be stored into the storage system according to the preset storage capacity of the preset fragments as storage granularity. The memory system of this embodiment may be a Key-Value (KV) memory system.

In this embodiment, in the process of reading the current data block, since the current data block is not stored in real time, but is stored according to the preset storage capacity of the preset fragment as the storage granularity, when the reading fails, there is data that has been read but the data amount does not reach the preset storage capacity of the preset fragment, but is not stored in the storage system, and at this time, the part of data may be stored in the cache first.

S204, determining the breakpoint position of the current data block which is failed to be read;

specifically, the breakpoint position of the current data block may be determined according to the data of the current data block that has been read. For example, during the process of reading a certain data block, the data block is stored for 3 times according to the preset storage capacity of the preset fragments as storage granularity. The preset storage capacity of the preset fragment is 50M, and the storage capacity of the data of the current data block stored in the cache is 6M, so that the breakpoint position of the current data block can be determined to be 156M.

S205, calculating a first fingerprint of the current data block based on the breakpoint position of the current data block;

specifically, all data before the breakpoint position of the current data block may be acquired based on the breakpoint position of the current data block, and then the first fingerprint of the current data block may be generated by adopting an encryption algorithm. In this embodiment, a Message Digest algorithm 5 (Message-Digest Algorithm 5; MD5), a secure hash algorithm 1 (Secure Hash Algorithm; SHA-1), or a 32-bit cyclic redundancy check (Cyclic Redundancy Check 32bit; CRC 32) algorithm may be specifically used to calculate the first fingerprint of the current data block according to all the data before the breakpoint position of the current data block. Further, the breakpoint location may be stored in the cache along with the first fingerprint of the current data block.

In this way, the first fingerprint of the current database can be calculated very accurately and efficiently.

S206, sending the breakpoint position of the current data block to the client side, and the first fingerprint of the current data block;

in this embodiment, the breakpoint position of the current data block and the first fingerprint of the current data block are sent to the client, so that the client calculates the second fingerprint of the current data block based on the local breakpoint position of the current data block; and determining data after continuously transmitting the breakpoint position of the current data block based on the first fingerprint of the current data block and the second fingerprint of the current data block.

The transmission in this embodiment may be based on a request from the client, or may be actively transmitted by the server after detecting a read failure. Specifically, on the client side, whether the file uploading fails is also detected, and if the file uploading fails, a breakpoint information request can be requested to the server. Correspondingly, the server receives the request, acquires the breakpoint position and the corresponding first fingerprint from the cache, and returns the breakpoint position and the corresponding first fingerprint to the client. The breakpoint location of this embodiment may also be referred to as offset.

Specifically, the client may obtain all data before the breakpoint position of the current data block based on the local breakpoint position of the current data block, and calculate the second fingerprint of the current data block according to MD5, SHA-1, or CRC32 algorithm. The same way as the calculation of the first fingerprint of the current data block is described above. Then detecting whether the first fingerprint of the current data block is consistent with the second fingerprint of the current data block; if the data of the current data block received by the server side are consistent, the data of the current data block can be determined to be very accurate, and the client side can continuously transmit the data of the current data block; otherwise, if the data of the current data block received by the server side are inconsistent, determining that the data of the current data block received by the server side are incorrect, and the client side is required to upload the data of the current data block again.

By adopting the method, the accuracy of the data received by the server side can be verified by the client side by detecting whether the first fingerprint of the current data block is consistent with the second fingerprint of the current data block, so that the accuracy of the data of the uploaded data block can be further and effectively ensured.

S207, if the client determines to continue transmitting the data of the current data block, receiving a continuous transmission data request sent by the client, wherein the continuous transmission data request carries the sequence identifier of the current data block, the continuous transmission identifier and the data after the breakpoint position of the current data block;

s208, acquiring data of the current data block, which is not stored in the storage system before the breakpoint position, from the cache;

s209, reading data after the breakpoint position of the current data block;

and S210, storing all the non-stored data in the current data block into a storage system by taking the preset storage capacity of the preset fragments as the storage granularity until the current data block is stored.

Specifically, in order to preserve the order of all the data in the current data block, the data of the current data block, which is not stored in the storage system before the breakpoint position, is first obtained from the cache, and then after the data, the data subsequently transferred at the breakpoint position is read. And stored together according to the storage granularity of the preset storage capacity of the preset fragments. That is, when data of a data block is stored in a storage system on the server side, the data is always stored with a predetermined granularity of slices. The bottom layer of the server can use a distributed KV storage system to store file data, and each KV data block corresponds to data with preset fragmentation granularity which is stored once.

It should be noted that, in the reading process of step S209, the process may also return to step S201 to perform reading detection, and according to the technical scheme of the above steps, after the reading fails, perform continuous transmission again until the current data block is stored, and determine that the current data block is successfully uploaded.

Optionally, the subsequent transmission request in step S207 may further carry a second fingerprint of the current data block; at this time, correspondingly, after step S207, before step S209, it may further include: based on the second fingerprint of the current data block and the first fingerprint of the current data block, it is determined to continue reading and storing data of the current data block.

In particular, whether the second fingerprint of the current data block is consistent with the first fingerprint of the current data block can be detected, and if so, the data of the current data block can be continuously read and stored. Otherwise, if the data block is inconsistent, the stored file can be considered to be uploaded in error, and at the moment, the server can return a response message of the uploading error to the client so that the client can upload the data of the current data block again.

By adopting the mode, the accuracy of uploading the current data block can be further ensured.

By adopting the scheme, the file uploading processing method can support breakpoint continuous uploading after uploading failure occurs in the process of uploading any data block of the target file, avoid uploading the data block again after uploading failure occurs, and further effectively improve the uploading success rate of the target file.

Further, in this embodiment, the data of the current data block is stored in the storage system with the preset storage capacity of the preset fragment as the storage granularity, so that the storage granularity can be effectively reduced, and the breakpoint position can be conveniently and efficiently and accurately located when the reading fails.

Further, in this embodiment, when the reading fails, the data of the current data block, the data amount of which does not reach the preset storage capacity of the preset fragment, may be stored in the cache, so that the data which is not stored may be efficiently and accurately stored, the data loss may be avoided, and further the accuracy and integrity of the data of the received data block may be effectively ensured.

Further, in this embodiment, the server may further verify the first fingerprint of the current data block at the server based on the second fingerprint information of the current data block in the continuous transmission data request of the client when the continuous transmission of the breakpoint begins, and perform continuous transmission of the breakpoint after the verification passes, so that accuracy of the uploaded data block can be effectively ensured, and further the success rate of file uploading can be effectively improved.

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure; as shown in fig. 3, the present embodiment provides a method for uploading a file, which may be applied to a client side to implement uploading processing of a large file, and specifically may include the following steps:

S301, in the process of uploading each data block in a plurality of data blocks of a target file to a server, if the current data block is detected and determined to be failed to be uploaded;

s302, acquiring the breakpoint position of the current data block when uploading fails;

s303, based on the breakpoint position of the current data block, continuing transmission of the data of the current data block is performed.

The difference from the embodiment shown in fig. 1 described above is that: the embodiment shown in fig. 1 describes the technical solution of the present disclosure on the server side, and the embodiment describes the technical solution of the present disclosure on the client side.

Specifically, when each data block of the target file is uploaded at the client side, in order to facilitate the uploading of the data block with larger storage capacity, a streaming reading mode is adopted, and the data block is read and uploaded at the same time. Moreover, during the uploading process, the client can detect whether the current data block fails to be uploaded. For example, if a response returned by the server is not received within a preset time period, the uploading is considered to be failed. Then, the breakpoint position of the current data block corresponding to the uploading failure can be obtained through interaction with the server, and further, the current data block can be continuously transmitted based on the breakpoint position.

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure; the present embodiment provides a method for uploading files, and further describes the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiment shown in fig. 3. As shown in fig. 4, the method for processing file uploading in this embodiment may specifically include the following steps:

s401, detecting and determining that the uploading of the current data block fails in the process of uploading each data block in a plurality of data blocks of a target file to a server;

s402 (S) sending a breakpoint information request to a server;

s403, receiving a breakpoint position of the current data block when uploading fails and returning by the server, and a first fingerprint of the current data block;

specifically, referring to the description of the embodiment shown in fig. 2, the server may obtain, from the cache, the breakpoint position of the current data block and the first fingerprint of the current data block, which are stored in advance, and return the first fingerprint to the server. Since the server is the data block receiving side, the breakpoint position of the current data block obtained from the server side is very accurate with respect to the breakpoint position of the current data block predicted by the client side.

S404, determining the data processing mode of the current data block to be continuous transmission of the data of the current data block based on the breakpoint position of the current data block and the first fingerprint of the current data block;

For example, when this step S404 is specifically implemented, the method may include the following steps:

(1) Acquiring data before the breakpoint position of the current data block based on the breakpoint position of the current data block and the locally stored data of the current data block;

(2) Calculating a second fingerprint of the current data block based on data preceding the breakpoint position of the current data block;

the calculation method of the first fingerprint of the current data block in the above embodiment may be referred to specifically, and the second fingerprint of the current data block is calculated, which is not described herein.

(3) Detecting whether the first fingerprint and the second fingerprint are identical; if the two are the same; executing the step (4); otherwise, if different, executing the step (5);

(4) And determining the data processing mode of the current data block to be that the data of the current data block is continuously transmitted.

(5) The method comprises the steps of determining that data before the breakpoint position of a current data block uploaded by a client side is inconsistent with data before the breakpoint position of the current data block received by a server side, further determining that uploading of the current data block fails, further determining that the data processing mode of the current data block is that data of the current data block is retransmitted, and effectively ensuring accuracy of the uploaded data of the data block.

Since both the first fingerprint and the second fingerprint are calculated using an encryption algorithm based on the data before the breakpoint location of the current data block. The first fingerprint is calculated by the server based on data before the breakpoint position of the received current data block at the server side, and the second fingerprint is calculated by the client based on data before the breakpoint position of the current data block stored locally at the client side.

S405, sending a continuous transmission data request to a server, wherein the continuous transmission data request carries the sequence identifier of the current data block, the continuous transmission identifier and the data after the breakpoint position of the current data block.

Optionally, the second fingerprint of the current data block may also be carried in the continuous data transmission request in step S405, so that the server may verify whether the received data of the current data block is correct again according to the second fingerprint of the current data block and the first fingerprint of the current data block. If the data is correct, the data which is continuously transmitted is further received, and the accuracy of the uploaded data block is further improved.

Further, in this embodiment, the client may further detect whether the first fingerprint of the server is correct based on the locally calculated second fingerprint of the current data block, and if so, determine to perform breakpoint continuous transmission, otherwise, determine to retransmit the current data block, so that accuracy of the uploaded data block can be effectively ensured, and further, the uploading success rate of the file can be effectively improved.

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure; the present embodiment provides a method for uploading files, which further introduces the technical solution of the present disclosure in more detail on the basis of the technical solution of the embodiments shown in fig. 1 to fig. 4. As shown in fig. 5, the method for processing file uploading in this embodiment may specifically include the following steps:

s501, the client side segments a target file to be uploaded into a plurality of data blocks according to a preset strategy; configuring sequence identification for each data block, and recording information of the data blocks in file meta information;

in this embodiment, the uploading of the target file is implemented by splitting the target file into a plurality of data blocks, and then interacting with different hypertext transfer protocol (Hyper Text Transfer Protocol; HTTP) requests.

The file meta information may include the size of the target file, the number of data blocks segmented by the target file, and the sequence identifier of each data block, for example, may be represented by a part number, and the size of each data block part. The part number of each data block can identify the location of the data block in the target file.

S502, the client interacts with the server, and sends file meta-information of the target file to be uploaded to the server;

s503, the client acquires a data block to be uploaded as a current data block;

in general, each data block may be uploaded sequentially in the order of the data blocks from front to back in the target file.

S504, the client side reads the data of the current data block in a streaming mode and uploads the data to the server;

in this embodiment, the client uploads the data of the current data block to the server based on the HTTP request.

S505, the server reads the data of the current data block according to the sequence from front to back; and detecting whether the reading is normal; if the reading is successful normally, storing the data to a storage system by taking the preset storage capacity of the preset fragments as storage granularity when the data with the preset storage capacity of the preset fragments are read;

In addition, alternatively, the fingerprint of the current data block may be calculated based on all the received data of the current data block every time the data of the preset storage capacity of the preset fragment is read. Because the data of the current data block is still read all the time, the fingerprint of the current data block is calculated in an intermediate state until the current data block is read, the fingerprint of the current data block is fixed, and the fingerprint of the current data block can be stored in a cache. The fingerprint of the current database may be stored in a cache so that the data received by the current data block may be checked in real time during transmission.

S506, if the read failure occurs, the server stores the read data of the current data block, the data quantity of which does not reach the preset storage capacity of the preset fragments, into a cache;

s507, the server determines the breakpoint position of the current data block which is failed to be read; calculating a first fingerprint of the current data block based on the breakpoint position of the current data block; storing the breakpoint position of the current data block and the first fingerprint of the current data block into a cache;

s508, when the client detects that the reading fails, sending a breakpoint information request to the server;

S509, the server acquires the breakpoint position of the current data block and the first fingerprint of the current data block from the cache and sends the breakpoint position and the first fingerprint of the current data block to the client;

s510, the client calculates a second fingerprint of the current data block based on the breakpoint position of the current data block;

specifically, referring to the description of the related embodiments, the client obtains, from the locally stored data of the current data block, the data before the breakpoint position of the current data block, based on the breakpoint position of the current data block; a second fingerprint of the current data block is then calculated based on the data preceding the breakpoint location of the current data block.

S511, the client detects whether the first fingerprint and the second fingerprint are identical; if the data block is the same, determining to continuously transmit the data of the current data block; otherwise, determining to retransmit the data of the current data block;

s512, the client sends a continuous transmission data request to the server, wherein the continuous transmission data request carries the sequence identifier of the current data block, the continuous transmission identifier, the second fingerprint of the current data block and the data after the break point position of the current data block;

for example, an identifier may be carried in the header information of the request to indicate whether to continue or retransmit, so that the server may perform corresponding processing.

If retransmission is needed, the client sends a request for retransmitting data to the server, and the request carries the sequence identifier and the retransmission identifier of the current data block. And begins to re-read the data of the current data block and re-upload in the manner of step 503. Correspondingly, on the server side, if a retransmission request is received, the data of the current data block stored in the storage system and the cache are required to be invalidated.

Or alternatively, the retransmission identifier or the continuous transmission identifier may not be carried. Since the data of the current data block is read from the beginning and the corresponding second fingerprint information does not exist at the time of retransmission, the corresponding second fingerprint information does not exist at the time of retransmission and does not need to be transmitted to the server. If the request carries the second fingerprint of the current data block, the request can be considered as the data of the continuous transmission current data block; otherwise, the data of the current data block is retransmitted.

S513, the server receives the continuous transmission data request and determines that the continuous transmission data is the data of the current data block; detecting whether a second fingerprint of a current data block in the continuous transmission data request is identical to a first fingerprint of the current data block;

s514, if the data are the same, the server acquires the data of the current data block, which are not stored in the storage system before the breakpoint position, from the cache; then, starting to read the data after the breakpoint position of the current data block;

In practical application, if the current data block is different, the server returns a response of the current data block uploading error to the client so that the client can retransmit the current data block.

S515, the server stores all the non-stored data in the current data block to be read into a storage system by taking the preset storage capacity of the preset fragments as storage granularity until the current data block is stored, and returns a response message of the stored current data block to the client;

specifically, in the storage process, when the server cannot read the data, the server can determine whether the current data block is stored according to the size of the current data block in the file meta-information and the size of the received data of the current data block. If the storage is finished, returning a response message of finishing the storage of the current data block to the client; otherwise, the reading is considered to be failed, and the process returns to the step S505 to perform the processing.

Optionally, the second fingerprint of the current data block may be carried when a response message is returned to the client that the current data block is stored. Correspondingly, the client can detect whether the second fingerprint is correct according to the first fingerprint of the local current data block. If the current data block is correct, determining that the current data block is completely uploaded. At this time, the client may return to step S502, and select to upload the next data block until all the data blocks are uploaded.

S516, after receiving all the data blocks of the target file, the server splices all the data blocks according to the sequence identification of each data block to obtain the target file.

All requests and responses between the client and the server in this embodiment are implemented using HTTP.

Fig. 6 is a schematic diagram of a relationship between a file, a data block, and a preset slice provided in the present disclosure. As shown in fig. 6, according to the manner of this embodiment, a complete file may be divided into a plurality of data blocks at the client according to the concept of multi-block upload (multi-part upload). That is, one file is composed of data blocks part greater than or equal to 1. When the server side is stored, the server side is stored according to slices (slices), and each slice is a preset slice and has a preset storage capacity. The preset storage capacity is of a fixed size, which may be, for example, 4MB, 8MB, 16MB, etc., but certainly not larger than the storage capacity of part. That is, one data block part may be composed of slices greater than or equal to 1. By setting smaller storage granularity at the server side, the storage of the data block can be more conveniently and effectively monitored, and the breakpoint position can be conveniently and accurately positioned in time when the reading fails.

According to the file uploading processing method, large file block uploading can be achieved through the scheme, granularity of file storage can be thinned from a data block part to a byte level, when the part uploading fails, more data can be transmitted to a server through each retransmission, and uploading success rate of the part is improved. And the breakpoint continuous transmission of byte level can effectively reduce the data volume of failed retransmission, greatly reduce the time consumption of failed retransmission while saving bandwidth, and effectively improve the success rate of file uploading.

FIG. 7 is a schematic diagram according to a sixth embodiment of the present disclosure; as shown in fig. 7, the present embodiment provides a server 700, including:

the detection module 701 is configured to detect and determine that data reading of a current data block fails in a process of receiving each data block in a plurality of data blocks of the target file uploaded by the client;

a determining module 702, configured to determine a breakpoint position of the current data block that fails to be read;

an obtaining module 703, configured to obtain data transmitted after the breakpoint position of the current data block based on the breakpoint position of the current data block.

The implementation principle and the technical effect of the server 700 of the present embodiment for implementing the uploading process of the file by using the above modules are the same as those of the above related method embodiments, and detailed description of the above related method embodiments may be referred to and will not be repeated here.

FIG. 8 is a schematic diagram according to a seventh embodiment of the present disclosure; as shown in fig. 8, this embodiment provides a server 800, which includes the modules with the same names and the same functions as those in fig. 7: a detection module 801, a determination module 802, and an acquisition module 803.

Further, as shown in fig. 8, the server 800 of the present embodiment further includes:

the storage module 804 is configured to store, in a storage system, the read data of the current data block with a preset storage capacity of a preset fragment as a storage granularity in a process of receiving each data block in a plurality of data blocks of the target file uploaded by the client;

the storage module 804 is further configured to store, in a cache, data of the current data block that has not reached the preset storage capacity of the preset fragment after detecting and determining that the data of the current data block fails to be read.

Further, as shown in fig. 8, in the server 800 of an embodiment of the present disclosure, further includes:

a calculating module 805, configured to calculate a first fingerprint of the current data block based on a breakpoint position of the current data block.

A sending module 806, configured to send, to the client, the breakpoint position of the current data block and the first fingerprint of the current data block, so that the client calculates, based on the local breakpoint position of the current data block, a second fingerprint of the current data block; and determining whether to continue transmitting data after the breakpoint position of the current data block based on the first fingerprint of the current data block and the second fingerprint of the current data block.

Further optionally, in an embodiment of the present disclosure, the obtaining module 803 is configured to:

receiving a continuous transmission data request sent by the client, wherein the continuous transmission data request carries a sequence identifier of the current data block, a continuous transmission identifier and data after the breakpoint position in the current data block;

acquiring data of the current data block, which is not stored in the storage system before the breakpoint position, from the cache;

reading the data after the breakpoint position in the current data block;

and storing all the non-stored data in the current data block to the storage system by taking the preset storage capacity of the preset fragments as storage granularity.

Further alternatively, in one embodiment of the present disclosure, the determining module 802 is further configured to:

based on the second fingerprint of the current data block and the first fingerprint of the current data block, determining to continue reading and storing data of the current data block.

The implementation principle and the technical effect of the server 800 of the present embodiment for implementing the uploading process of the file by using the above modules are the same as those of the above related method embodiments, and detailed description of the above related method embodiments may be referred to and will not be repeated here.

FIG. 9 is a schematic diagram according to an eighth embodiment of the present disclosure; as shown in fig. 9, the present embodiment provides a client device 900, including:

the detection module 901 is configured to detect and determine that an upload failure of a current data block occurs during uploading each data block of a plurality of data blocks of a target file to a server;

an obtaining module 902, configured to obtain a breakpoint position of the current data block corresponding to an upload failure;

and the transmission module 903 is configured to continuously transmit the data of the current data block based on the breakpoint position of the current data block.

The implementation principle and the technical effect of the client device 900 in this embodiment for implementing the uploading process of the file by using the above modules are the same as those of the above related method embodiments, and detailed description of the above related method embodiments may be referred to and will not be repeated here.

FIG. 10 is a schematic diagram according to a ninth embodiment of the present disclosure; as shown in fig. 10, this embodiment provides a client device 1000, including the modules with the same names and the same functions as those in fig. 9 described above: a detection module 1001, an acquisition module 1002 and a transmission module 1003.

Further, in the client device 1000 of the present embodiment, the obtaining module 1002 is configured to:

and acquiring the breakpoint position of the current data block when the uploading fails from the server.

Further optionally, in an embodiment of the present disclosure, the obtaining module 1002 is configured to:

and acquiring a first fingerprint of the current data block from the server when uploading fails, wherein the first fingerprint of the current data block is calculated by the server based on the breakpoint position of the current data block.

As shown in fig. 10, in one embodiment of the present disclosure, the client device 1000 further includes:

a determining module 1004, configured to determine, based on the breakpoint position of the current data block and the first fingerprint of the current data block, that the data processing manner of the current data block is to continuously transmit the data of the current data block.

Further optionally, in an embodiment of the present disclosure, the determining module 1004 is configured to:

Acquiring data before the breakpoint position of the current data block based on the breakpoint position of the current data block and locally stored data of the current data block;

calculating a second fingerprint of the current data block based on data preceding the breakpoint position of the current data block;

detecting and determining that the first fingerprint and the second fingerprint are identical;

and determining the data processing mode of the current data block to be continuous transmission of the data of the current data block.

Further optionally, in an embodiment of the present disclosure, the transmission module 1003 is configured to:

and sending a continuous transmission data request to the server, wherein the continuous transmission data request carries the sequence identifier of the current data block, the continuous transmission identifier and the data after the breakpoint position in the current data block.

Further optionally, in an embodiment of the present disclosure, the determining module 1004 is further configured to:

and if the first fingerprint is different from the second fingerprint, determining that the data processing mode of the current data block is to retransmit the data of the current data block.

The implementation principle and the technical effect of the client device 1000 in this embodiment for implementing the uploading process of the file by using the above modules are the same as those of the above related method embodiments, and detailed description of the above related method embodiments may be referred to and will not be repeated here.

FIG. 11 is a schematic diagram according to a tenth embodiment of the present disclosure; as shown in fig. 11, this embodiment further provides a file uploading processing system 1100, which includes a client device 1101 and a server 1102, where the client device 1101 is communicatively connected to the server 1102, to implement uploading each data block of a plurality of data blocks of a target file to the server 1102; the server 1102 of the present embodiment may be a server as shown in fig. 7 or fig. 8, the client device 1101 may be a client device as shown in fig. 9 or fig. 10, and specifically, the method for uploading files as shown in fig. 1-5 may be used to implement the uploading process of the files.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 12 shows a schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 12, the apparatus 1200 includes a computing unit 1201, which may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a Random Access Memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 may also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

Various components in device 1200 are connected to I/O interface 1205, including: an input unit 1206 such as a keyboard, mouse, etc.; an output unit 1207 such as various types of displays, speakers, and the like; a storage unit 1208 such as a magnetic disk, an optical disk, or the like; and a communication unit 1209, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks.

The computing unit 1201 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1201 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the methods described above of the present disclosure. For example, in some embodiments, the above-described methods of the present disclosure may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 12012. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1200 via ROM 1202 and/or communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the above-described method of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the above-described methods of the present disclosure in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A file uploading processing method comprises the following steps:

in the process of receiving each data block in a plurality of data blocks of a client-side uploading target file, detecting and determining that the data of the current data block is failed to read;

2. The method of claim 1, wherein in receiving each of a plurality of data blocks of the target file uploaded by the client, the method comprises:

storing the read data of the current data block to a storage system by taking the preset storage capacity of the preset fragments as storage granularity;

if the data of the current data block is detected and determined to be read out failed, the method further comprises the following steps:

and storing the read data of which the data quantity does not reach the preset storage capacity of the preset fragments in a cache, wherein the data of the current data block does not reach the preset storage capacity of the preset fragments.

3. The method of claim 2, wherein after determining the breakpoint location of the current data block that failed to read, the method further comprises:

a first fingerprint of the current data block is calculated based on the breakpoint location of the current data block.

4. A method according to claim 3, wherein after computing the first fingerprint of the current data block based on the breakpoint location of the current data block, the method further comprises:

sending the breakpoint position of the current data block and the first fingerprint of the current data block to the client so that the client calculates a second fingerprint of the current data block based on the local breakpoint position of the current data block; and determining whether to continue transmitting data after the breakpoint position of the current data block based on the first fingerprint of the current data block and the second fingerprint of the current data block.

5. The method according to claim 3 or 4, wherein acquiring data subsequently transferred at the breakpoint location of the current data block based on the breakpoint location of the current data block comprises:

reading the data after the breakpoint position in the current data block;

6. The method of claim 5, wherein the retransmission request further carries a second fingerprint of the current data block; before retrieving, from the cache, data of the current block of data that was not previously stored to the storage system at the breakpoint location, the method further includes:

7. A file uploading processing method comprises the following steps:

8. The method of claim 7, wherein obtaining the breakpoint location of the current data block at the time of the upload failure comprises:

9. The method of claim 8, wherein the method further comprises:

Wherein, based on the breakpoint position of the current data block, before the data of the current data block is continuously transmitted, the method further includes:

and determining a data processing mode of the current data block based on the breakpoint position of the current data block and the first fingerprint of the current data block, wherein the data processing mode is to continuously transmit the data of the current data block.

10. The method of claim 9, wherein determining, based on the breakpoint location of the current data block and the first fingerprint of the current data block, that the data of the current data block is processed in a manner that includes:

11. The method of claim 10, wherein continuing the data of the current data block based on the breakpoint location of the current data block comprises:

12. The method of claim 10, wherein the method further comprises:

13. A server, comprising:

the detection module is used for detecting and determining the data reading failure of the current data block in the process of receiving each data block in a plurality of data blocks of the target file uploaded by the client;

14. The server of claim 13, further comprising:

the storage module is used for storing the read data of the current data block into the storage system by taking the preset storage capacity of the preset fragments as the storage granularity in the process of receiving each data block in a plurality of data blocks of the target file uploaded by the client;

and the storage module is further used for storing the read data of which the data quantity does not reach the preset storage capacity of the preset fragments in a cache after detecting and determining that the data of the current data block fails to be read.

15. The server of claim 14, further comprising:

and the calculating module is used for calculating the first fingerprint of the current data block based on the breakpoint position of the current data block.

16. The server of claim 15, further comprising:

the sending module is used for sending the breakpoint position of the current data block and the first fingerprint of the current data block to the client so that the client can calculate the second fingerprint of the current data block based on the local breakpoint position of the current data block; and determining whether to continue transmitting data after the breakpoint position of the current data block based on the first fingerprint of the current data block and the second fingerprint of the current data block.

17. The server according to claim 15 or 16, wherein the acquisition module is configured to:

Reading the data after the breakpoint position in the current data block;

18. The server of claim 17, wherein the determination module is further configured to:

19. A client device, comprising:

20. The client device of claim 19, wherein the acquisition module is configured to:

21. The client device of claim 20, wherein the acquisition module is configured to:

The client device further includes:

and the determining module is used for determining that the data processing mode of the current data block is to continuously transmit the data of the current data block based on the breakpoint position of the current data block and the first fingerprint of the current data block.

22. The client device of claim 21, wherein the determining module is configured to:

23. The client device of claim 22, wherein the transmission module is configured to:

24. The client device of claim 22, wherein the determining module is further configured to:

25. The file uploading processing system comprises a client device and a server, wherein the client device is in communication connection with the server to upload each data block in a plurality of data blocks of a target file to the server; the server employing a server according to any of the preceding claims 13-18 and the client device employing a client device according to any of the preceding claims 19-24.

26. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6 or 7-12.

27. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-6 or 7-12.

28. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-6 or 7-12.