CN112583904A - File uploading method, device, equipment and storage medium - Google Patents
File uploading method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112583904A CN112583904A CN202011406833.XA CN202011406833A CN112583904A CN 112583904 A CN112583904 A CN 112583904A CN 202011406833 A CN202011406833 A CN 202011406833A CN 112583904 A CN112583904 A CN 112583904A
- Authority
- CN
- China
- Prior art keywords
- data block
- target file
- data
- file
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a file uploading method, device, equipment and storage medium, and relates to the technical field of cloud computing and cloud storage. The specific implementation scheme is as follows: receiving an uploading request corresponding to a target file from a terminal device; determining the size of a target block corresponding to a target file according to the current network parameters, and sending the size of the target block to terminal equipment, wherein the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameters, and the size of the target block is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the size of the target block; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. According to the scheme, the file uploading success rate is improved.
Description
Technical Field
The present application relates to cloud computing and cloud storage technologies in the internet field, and in particular, to a method, an apparatus, a device, and a storage medium for uploading a file.
Background
With the development and popularization of cloud storage technology, more and more users prefer to store files in a cloud server. In practical application, when a file is large, the uploading success rate of the file is low.
In the prior art, in order to ensure successful uploading of a large file, a cloud server provides a block uploading interface, so that a terminal device divides the large file into a plurality of data blocks and sends the data blocks to the cloud server one by one. In order to facilitate a subsequent file query process, the cloud server needs to generate and store file meta information after receiving the data blocks, and identify the file meta information through the indexes.
However, in the above manner, the number of blocks cannot be increased without limit due to the index structure of the file meta-information, so that the cloud server limits the maximum number of blocks in the block uploading interface. Thus, when dealing with a very large file, even if the block is divided according to the maximum block number, the size of each data block may still be large, and when the network environment is poor, the success rate of file uploading is still low.
Disclosure of Invention
The application provides a file uploading method, a file uploading device, file uploading equipment and a storage medium.
In a first aspect, the present application provides a file uploading method, including:
receiving an uploading request corresponding to a target file from a terminal device;
determining the size of a target block corresponding to the target file according to the current network parameters, and sending the size of the target block to the terminal equipment; the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal device to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the target block size;
and receiving the data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure.
In a second aspect, the present application provides a file uploading apparatus, including:
the receiving and sending module is used for receiving an uploading request corresponding to the target file from the terminal equipment;
the determining module is used for determining the size of a target block corresponding to the target file according to the current network parameters;
the transceiver module is further configured to send the size of the target block to the terminal device; the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal device to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the target block size;
the transceiver module is further configured to receive the plurality of data blocks from the terminal device;
and the storage module is used for storing the meta information of the target file by adopting a single-layer index structure.
In a third aspect, the present application provides an electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the first aspects.
In a fourth aspect, the present application provides a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of the first aspects.
In a fifth aspect, the present application provides a computer program product comprising computer instructions which, when executed by a processor, implement the method of any one of the first aspects.
The application provides a file uploading method, a file uploading device, equipment and a storage medium, wherein the method comprises the following steps: receiving an uploading request corresponding to a target file from a terminal device; determining the size of a target block corresponding to a target file according to the current network parameters, and sending the size of the target block to terminal equipment, wherein the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameters, and the size of the target block is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the size of the target block; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. In the process, the limitation on the maximum block number is cancelled by storing the meta-information of the target file by adopting a single-layer index structure, so that the purpose of reducing the data volume uploaded every time as much as possible can be realized. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, the data blocks obtained by dividing the target file can be successfully uploaded, and the success rate of uploading the file is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
fig. 1 is a schematic diagram of a file upload application scenario provided in the present application;
FIG. 2 is a schematic diagram of file chunking;
FIG. 3 is a diagram illustrating an index structure of meta information of a file in the prior art;
fig. 4 is a schematic flowchart of a file uploading method provided in the present application;
FIG. 5 is a diagram illustrating an index structure of meta information of a file provided in the present application;
FIG. 6 is a schematic flow chart illustrating another file uploading method provided by the present application;
fig. 7 is a schematic structural diagram of a document uploading apparatus provided in the present application;
fig. 8 is a schematic structural diagram of another document uploading device provided in the present application;
fig. 9 is a schematic structural diagram of an electronic device provided in the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The application provides a file uploading method, device, equipment and storage medium, which are applied to the technical fields of cloud storage, cloud computing and the like in the field of internet so as to improve the file uploading success rate.
Fig. 1 is a schematic diagram of a file upload application scenario provided in the present application. As shown in fig. 1, the application scenario includes: terminal equipment and high in the clouds server. The terminal equipment is connected with the cloud server through a network.
The terminal device can be an electronic device with information processing capability, such as a smart phone, a tablet computer or a desktop computer. The terminal device can access the cloud server through a webpage or a client side, and sends a file uploading request to the cloud server so as to achieve the purpose of uploading files to the cloud server. The terminal equipment can also access the cloud server in a webpage or client-side mode, and sends a file query request or a file download request to the cloud server so as to achieve the purpose of querying/downloading files from the cloud server.
The cloud server of this embodiment may be, for example, a network disk server that provides a network disk storage service to a user. The cloud server may employ distributed storage technology, or may be a server incorporating a blockchain. Of course, the cloud server may also employ other storage technologies.
And after receiving a file uploading request sent by the terminal equipment, the cloud server receives a file to be uploaded and stores the file. After receiving a file downloading request sent by the terminal equipment, the cloud server inquires the requested file and sends the file to the terminal equipment. After receiving a file query request sent by the terminal equipment, the cloud server queries a file of the request and returns related information of the file to the terminal equipment.
The cloud server provides general cloud storage service for the user. "Universal" means that a user can store any type of file, any size. Moreover, any file should be stored and accessed to provide the same service guarantee for the user, for example, the file uploading success rate.
However, in practical applications, the larger the file, the lower the upload success rate. The main reasons are that: when the file is larger, the network interaction time between the terminal device and the cloud server is longer, so that the probability of data transmission errors on the network is higher. Even if the network protocol layer and the service layer have the failure retransmission strategy, successful uploading cannot be guaranteed in a hundred percent.
In order to ensure successful uploading of larger files, the cloud server provides a block uploading interface. The terminal device divides a large file into a plurality of data blocks, and calls the uploading interface once for each data block so as to send the data block to the cloud server. The large file is divided into the data blocks, so that the data volume transmitted between the terminal equipment and the cloud server during each interaction is small, and the uploading success rate of the file is improved to a certain extent.
In order to facilitate understanding of the technical solution of the present application, the following first describes a file block uploading process with reference to fig. 2 and 3.
The cloud server provides three uploading interfaces for the terminal device, namely an initial uploading interface, a block uploading interface and a submission uploading interface.
When the terminal equipment receives a file uploading operation instruction input by a user, the terminal equipment sends a file uploading request to the cloud server by calling the initial uploading interface. And after receiving the file uploading request, the cloud server generates a globally unique index identifier for the file. On one hand, the cloud server informs the terminal equipment of the index identification of the file, so that the terminal equipment can associate the index identification of the file with the block transmission process of the file. On the other hand, the cloud server can also store the index identifier of the file into the meta information of the file, so that the file can be queried through the index identifier of the file in subsequent file querying, downloading and other processes.
After the terminal device obtains the index identifier of the file from the cloud server, the file to be uploaded can be divided into a plurality of data blocks. FIG. 2 is a schematic diagram of file chunking. As shown in fig. 2, for example, it is assumed that a file is divided into data block 0, data block 1, data block 2, … …, and data block n. And the terminal equipment calls a block uploading interface for each data block once so as to upload the data block to the cloud server.
Correspondingly, after receiving each data block, the cloud server performs fragment storage on the data block. Specifically, the cloud server randomly allocates an index identifier to each data chunk, divides each data chunk into a plurality of data pieces according to the fragment size required by the storage system of the cloud server (as shown in fig. 2, divides data chunk 0 into data piece 0, data piece 1, data piece 2, … …, and data piece m), and also randomly allocates an index identifier to each data piece. And after the cloud server stores each data block, storing the meta information of the data block. The meta-information of the data block includes an index identifier of the data block, fingerprint information, and index identifiers of data pieces in the data block.
After the terminal device finishes uploading all the data blocks, the terminal device calls the submission uploading interface to inform the cloud server that all the data blocks of the file are uploaded completely. The cloud server organizes the meta information of all the data blocks corresponding to the file into the meta information of the file, and stores the meta information of the file, so that the uploading process of the whole file is completed.
Fig. 3 is a diagram illustrating an index structure of meta information of a file in the prior art. As shown in fig. 3, the meta information of the file stored in the cloud server includes: the index identification of the file, the size of the file, the fingerprint information of the file, and the like, and the meta information of the file also stores the index identification list of all data blocks. Therefore, when the user subsequently queries the file, the cloud server can acquire the index identification of each data block corresponding to the file according to the stored meta information of the file. According to the index identification of each data block, the meta-information of each data block can be further acquired, and the index identification of each data piece corresponding to the data block is acquired from the meta-information of each data block, so that the content of the file is inquired.
As can be seen from the above process, in order to facilitate the file query/download process, the cloud server needs to adopt an index structure (e.g., the index structure shown in fig. 3) with at least two layers to store the file information.
The cloud server needs to store the index identifiers of all the data blocks in the meta information of the file, so that the number of the blocks of the file cannot be increased without limit due to the storage mode. Therefore, the maximum number of blocks is usually limited in the block uploading interface provided by the cloud server. For example, some cloud servers limit the maximum number of blocks to 10000, and some cloud servers limit the maximum number of blocks to 2000. This limitation of the number of partitions also limits the minimum of the partition size. For example, if the size of the file to be uploaded is 10GB, and the maximum number of blocks is 2000 blocks, the size of each data block is 5MB (here, 1GB is 1000 MB). If the size of the file to be uploaded is 100GB, the size of each data block is 50MB if the maximum number of blocks is 2000. When the network environment is poor, the success rate of transmitting 50MB of data blocks in a single request is also not guaranteed. According to on-line actual data statistics, when a file is larger than 10GB, the file uploading failure rate may reach more than 10%, and user experience is seriously affected.
It can be seen that in the above prior art, when dealing with very large files, the size of each data block may still be large (e.g., greater than 10MB, or even greater than 50MB) even if the division is performed according to the maximum number of blocks. Under the condition of poor network environment, the uploading success rate of the files is still low even if the blocking transmission process is adopted.
In addition, in the above prior art, since a large amount of meta information of data blocks needs to be stored in the meta information of a file, the capacity of the meta information of the file is increased, and on one hand, a large storage space needs to be occupied, and on the other hand, when the file needs to be queried, the amount of computation required for analyzing the meta information of the file is also large, which results in low file querying efficiency.
In order to solve at least one of the above technical problems, the present application provides a file uploading method, which has a core idea of reducing the amount of data uploaded each time as much as possible, that is, dividing the data blocks of a file as small as possible. For the same file, the smaller the divided data blocks are, the greater the number of corresponding data blocks is. Therefore, the core problem to be solved by the present application is how to break the limitation on the maximum number of blocks in the prior art. According to the file uploading method, the limit on the maximum block number is cancelled by improving the index structure of the meta-information of the file, so that the purpose of reducing the data volume uploaded every time as much as possible can be achieved. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, the data blocks obtained by dividing the target file can be successfully uploaded, and the success rate of uploading the file is improved.
The technical solution of the present application will be described in detail with reference to several specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 4 is a schematic flowchart of a file uploading method provided by the present application. The method of the embodiment may be performed by a cloud server. As shown in fig. 4, the method of this embodiment may include:
s401: and receiving an uploading request corresponding to the target file from the terminal equipment.
Specifically, when the terminal device receives a file uploading instruction input by a user, the terminal device determines a target file to be uploaded, and sends an uploading request to the cloud server to request to upload the target file. The target file to be uploaded includes but is not limited to: text files, audio files, video files, etc.
For example, the terminal device may send an upload request corresponding to the target file by calling an initial upload interface provided by the cloud server. And the cloud server receives an uploading request corresponding to the target file.
Optionally, after receiving the upload request, the cloud server may generate a globally unique index identifier for the target file. The cloud server can also inform the terminal equipment of the index identification of the target file, so that the terminal equipment can associate the index identification of the target file with the block transmission process of the file. For example, when each data block is uploaded, the terminal device carries the index identifier of the target file to indicate that the currently transmitted data block is the data block in the target file.
The index identifier of the target file can also be used for storing into the meta information of the target file, so that the target file can be queried through the index identifier in subsequent file query, download and other processes.
S402: and determining the size of a target block corresponding to the target file according to the current network parameters, and sending the size of the target block to the terminal equipment, wherein the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameters, and the size of the target block is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the size of the target block.
One difference between the present embodiment and the prior art is that, in the present embodiment, when determining the size of the target chunk corresponding to the target file, the limitation of the maximum chunk number is not considered, and the smaller each data chunk is divided as much as possible, so as to ensure successful upload of each data chunk. For example, the cloud server may configure a default smaller chunk size (e.g., 1MB) to ensure that the data chunk can be successfully uploaded in the weak network environment.
Of course, the smaller the block size, the more IO times, and the throughput will be reduced accordingly. Therefore, in the embodiment of the application, in order to balance the success rate and the throughput rate, the cloud server may determine the size of the target block according to the current network parameter. That is to say, the cloud server determines the size of the target block that can be uploaded successfully in the current network environment.
After receiving an upload request corresponding to a target file, the cloud server can obtain current network parameters according to the upload request. For example, when the terminal device sends an upload request, the upload request may carry the current network parameters. Or the cloud server measures the current network parameters according to the receiving condition of the uploading request. Or after receiving the uploading request, the cloud server acquires the current network parameters through a message interaction process with the terminal equipment. This embodiment is not limited to this.
In this embodiment, the network parameter may be any parameter for indicating the quality of the network environment. Optionally, the network parameters may include one or more of the following: the position information of the terminal equipment, the operator information accessed by the terminal equipment, the network transmission rate, the network signal intensity and the like.
When the network environment quality indicated by the current network parameters is good, the size of the target block can be set to be a larger value; when the network environment quality indicated by the current network parameter is poor, the target block size may be set to a smaller value. By dynamically determining the size of the target block according to the current network parameters, higher throughput rate can be ensured on the premise of ensuring the successful uploading of the file, and the uploading efficiency is improved.
In a possible implementation manner, the block size thresholds corresponding to different network parameters may be determined according to a historical file uploading condition or according to a file uploading test condition. The block size threshold corresponding to each network parameter indicates that: in the network environment corresponding to the network parameters, the maximum block size of successful uploading can be ensured. For example, if the block size threshold corresponding to a certain network parameter is 10MB, it indicates that successful upload of a maximum data block of 10MB can be guaranteed in the network environment corresponding to the network parameter, and if the data block is greater than 10MB, successful upload cannot be guaranteed.
When the cloud server determines the target block size corresponding to the target file according to the current network parameter, the cloud server may use the block size threshold corresponding to the current network parameter as the target block size. Alternatively, the target chunk size is set to some value less than the chunk size threshold. For example, if the block size threshold corresponding to the current network parameter is 10MB, the cloud server may determine the target block size to be 10MB, 8MB, or 5MB, etc.
After determining the size of a target block corresponding to the target file, the cloud server sends the size of the target block to the terminal device, so that the terminal device divides the target file into a plurality of data blocks according to the size of the target block. Wherein the size of each data block is less than or equal to the target block size.
In this embodiment, the size of the target block corresponding to the target file is determined according to the current network parameter, so that the determined size of the target block meets the current network environment, and it is ensured that all data blocks obtained by dividing the target file can be successfully uploaded, thereby improving the success rate of file uploading.
S403: and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure.
After the terminal device divides the target file into a plurality of data blocks, each data block can be uploaded to the cloud server through the block uploading interface. And after receiving each data block, the cloud server performs fragment storage on each data block. And after the cloud server receives the data blocks, generating and storing the meta information of the target file.
Another difference between the present embodiment and the prior art is that the present embodiment does not use the index structure of at least two layers in the prior art to store the meta information of the target file, but uses the index structure of a single layer to store the meta information of the target file. It should be understood that the "single-layer index structure" herein means that only information at the file level is stored in the meta information of the target file, and the index identifier of each data block is not stored. That is, the meta information of each data block is no longer stored in the meta information of the target file in the present embodiment.
Fig. 5 is a schematic diagram of an index structure of file meta information provided in the present application. As shown in fig. 5, only information at the file level is stored in the meta information of the target file, for example: the size of the file, the index identification of the file, the fingerprint information of the file, etc. The difference from the file meta information shown in fig. 3 is that the index identification of each data block is not stored.
In this embodiment, the phrase "not storing the index identifier of each data block" means that the index identifier of each data block is not explicitly stored. Of course, in order to index the information of each data block of the target file in the subsequent file query process, the index identifier of each data block may be implicitly stored in the meta information of the target file, or the index identifier of each data block may be implicitly indicated. The present embodiment does not limit the manner of implicit storage or implicit indication, and reference may be made to the detailed description of the subsequent embodiments for a possible implementation manner.
Since the index identifier list of the data block is no longer stored in the meta-information of the target file in this embodiment, the meta-information of the target file is not affected by the number of the blocks, and thus the present embodiment does not need to limit the maximum number of the blocks. That is, the present embodiment stores the meta information of the target file by using a single-layer index structure to cancel the limitation on the maximum number of chunks, thereby enabling "reducing the data amount uploaded at each time as much as possible" to be implemented.
The file uploading method provided by the embodiment comprises the following steps: receiving an uploading request corresponding to a target file from a terminal device; determining the size of a target block corresponding to a target file according to the current network parameters, and sending the size of the target block to terminal equipment, wherein the size of the target block is smaller than or equal to a block size threshold corresponding to the current network parameters, and the size of the target block is used for enabling the terminal equipment to divide the target file into a plurality of data blocks, and the size of each data block is smaller than or equal to the size of the target block; and receiving a plurality of data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure. In the process, the limitation on the maximum block number is cancelled by storing the meta-information of the target file by adopting a single-layer index structure, so that the purpose of reducing the data volume uploaded every time as much as possible can be realized. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, the data blocks obtained by dividing the target file can be successfully uploaded, and the success rate of uploading the file is improved.
On the basis of any of the above embodiments, the following describes the technical solution of the present application in more detail with reference to fig. 6.
Fig. 6 is a schematic flowchart of another file uploading method provided by the present application. As shown in fig. 6, the method of the present embodiment includes:
s601: and the terminal equipment sends an uploading request corresponding to the target file to the cloud server.
Correspondingly, the cloud server receives an uploading request corresponding to the target file from the terminal equipment.
S602: and the cloud server determines the size of a target block corresponding to the target file according to the current network parameters. And the target block size is smaller than or equal to the block size threshold corresponding to the current network parameter.
Optionally, after receiving the upload request corresponding to the target file, the cloud server may further determine whether the upload request is a first upload request for the target file. That is, it is determined whether the target file is uploaded for the first time or is re-uploaded.
In some scenarios, if it is determined that the upload request is the first upload request for the target file, S602 is performed. That is, if the target file is determined to be uploaded for the first time, the current network parameters are obtained, and the size of the target block corresponding to the target file is determined according to the current network parameters.
In other scenarios, if it is determined that the upload request is not the first upload request for the target file, it indicates that the target file is to be re-uploaded. In this case, the historical chunk size corresponding to the target file may be obtained, and the historical chunk size may be adjusted to obtain the target chunk size corresponding to the target file. For example, the historical chunk size may be adjusted to a smaller value, and the adjusted chunk size may be used as the target chunk size.
In this embodiment, if the target file is uploaded for the first time, and the target block size is set to be larger, which results in an uploading failure, when the terminal device initiates a re-upload, the cloud server may issue a smaller target block size to the terminal device (for example, the target block size during the re-upload may be half of the corresponding block size during the first upload), so as to ensure that the file is successfully uploaded.
S603: and the cloud server sends the size of the target block to the terminal equipment.
S604: and the terminal equipment divides the target file into a plurality of data blocks according to the size of the target block, wherein the size of each data block is smaller than or equal to the size of the target block.
S605: and the terminal equipment respectively sends each data block to the cloud server.
Correspondingly, the cloud server receives each data block from the terminal device.
S606: and the cloud server stores each data block according to the index identification of the target file, and generates and caches the meta information of each data block.
For any first data block in the plurality of data blocks, the cloud server may store the first data block in a feasible manner as follows, and generate and cache meta information of the first data block.
(1) And determining the index identification of the first data block according to the index identification of the target file.
Optionally, when the terminal device sends each data block to the cloud server, the sequence number of the data block is carried. The cloud server can calculate the index identification of the target file and the sequence number of each data block by adopting a first hash operation to determine the index identification of the data block. As shown in the following equation:
PartIDi＝Func1(FileID，i)
wherein i is the serial number of the data block, PartIDiThe index id of the ith data block is the index id of the target file, and Func1() is the first hash operation.
(2) The method comprises the steps of dividing a first data block into a plurality of data pieces, determining an index identification of each data piece according to the index identification of the first data block, and storing the data pieces according to the index identifications of the data pieces.
Optionally, the cloud server may divide the first data block into a plurality of data pieces according to a size of the pieces required by the storage system of the cloud server. And the cloud server calculates the index identification of the first data block and the sequence number of each data slice by adopting a second hash operation to determine the index identification of each data slice. As shown in the following equation:
SliceIDj＝Func2(PartIDi，j)
wherein j is the serial number of the data slice, SliceIDjFor the index identification, PartID, of the jth data slice in the ith data blockiFunc2() is the second hash operation, identified as the index of the ith data block.
(3) And generating and caching the meta-information of the first data block, wherein the meta-information of the first data block comprises the index identification and the fingerprint information of the first data block.
The embodiment is different from the prior art in that after the cloud server generates the meta-information of the data block, the meta-information of the data block is not persistently stored (i.e., the meta-information of the data block is not stored in the meta-information of the target file), but the meta-information of the data block is cached in the cache system. For example, the cache system may be a redis cache system. The meta information of the cached data block can be used to verify whether the target file is successfully uploaded.
S607: and the terminal equipment sends a file submission request to the cloud server.
Correspondingly, the cloud server receives a file submission request from the terminal device. The file submission request is used for informing the cloud server that all data blocks of the target file are uploaded. Optionally, the terminal device may send a file submission request to the cloud server by calling a submission uploading interface.
S608: the cloud server generates and stores the meta information of the target file when verifying that the data blocks are uploaded correctly according to the meta information of the data blocks.
And the cloud server checks whether all the data blocks of the target file are uploaded correctly according to the meta information of the plurality of data blocks cached in the cache system. And if all the data blocks of the target file are correctly uploaded, generating and storing the meta information of the target file. The meta information of the target file adopts a single-layer index structure, and the meta information of the target file includes the index identifier of the target file and does not include the meta information of each data block in the target file (i.e., does not include the index identifier of each data block in the target file).
S609: and the cloud server clears the cached meta information of the data blocks.
In this embodiment, the index identifier of each data block is determined according to the index identifier of the target file and the first hash operation, and the index identifier of each data slice is determined according to the index identifier of the data block and the second hash operation, so that the index identifiers of each data block and each data slice corresponding to the target file can be calculated only according to the index identifier of the target file, that is, the content of the target file can be obtained by indexing only according to the index identifier of the target file. Therefore, the storage of the meta-information of the target file by adopting a single-layer index structure is realized, so that the limitation on the maximum block number is eliminated, and the aim of reducing the data volume uploaded every time as much as possible can be realized. Furthermore, the size of the target block corresponding to the target file is determined according to the current network parameters, so that the determined size of the target block meets the current network environment, the data blocks obtained by dividing the target file can be successfully uploaded, and the success rate of uploading the file is improved.
On the basis of the embodiment shown in fig. 6, the following describes the query flow of the target file.
When the terminal equipment receives a file query operation input by a user, the terminal equipment determines a target file to be queried and sends a file query request corresponding to the target file to the cloud server. The file query request is used for requesting to query the content of the target file. And the cloud server acquires the index identification of the target file according to the meta information of the target file. The cloud server obtains index identifications of a plurality of data blocks corresponding to the target file according to the index identification of the target file and the first hash operation. Further, the cloud server obtains the index identifications of the data pieces corresponding to each data block according to the index identification of each data block and the second hash operation. Furthermore, the cloud server may obtain the content of the target file from the storage space corresponding to the index identifier of each data slice. Through the process, the query process of the target file is realized.
In the embodiment, the meta information of each data block is not stored in the meta information of the target file any more, so that the storage capacity of the meta information of the file is reduced, the analysis calculation amount of the meta information of the file during file query is reduced, and the file query efficiency is improved.
In practical application, a phenomenon that a certain data block is uploaded again may exist in a file uploading process. For example, a file is partially changed during the uploading process, and the changed data block may be re-uploaded. On the basis of the embodiment shown in fig. 6, the index identifier of the data block is obtained by calculating the index identifier of the target file and the sequence number of the data block by using a first hash operation. Thus, when a certain data block is re-uploaded, the index identification of the re-uploaded data block and the index identification of the previously uploaded data block are the same (because the calculation formula of the index identification of the data block is the same).
If the cloud server adopts a distributed storage system, since the distributed storage system cannot guarantee the write-in time sequence of data, when a certain data is repeatedly received, an overwriting mode is usually adopted. Thus, when the data block n is uploaded again, it may happen that the new data block n is written first, and the old data block n is written later and overwrites the new data block n, resulting in file data confusion.
In this embodiment, to avoid the problem of data confusion caused by re-uploading data blocks, the cloud server may adopt a locking mode when receiving each data block and storing the data block.
In a possible implementation manner, after the cloud server receives the first data block and determines the index identifier of the first data block, it may also determine whether the first data block is received for the first time according to the index identifier of the first data block. For example, it may be queried whether the meta information of the first data block exists in the cache system, and if the meta information exists, it indicates that the first data block is not received for the first time, and if the meta information does not exist, it indicates that the first data block is received for the first time.
If it is determined that the first data block is received for the first time, the first data block may be stored according to the procedure of the above embodiment.
And if the first data block is determined not to be received for the first time, comparing the content of the first data block with the content of the second data block. The second data block is the data block which is received for the first time and has the same index identification with the first data block. And if the comparison result is consistent, the first data block is considered to be uploaded successfully. And if the comparison result is inconsistent, the first data block is considered to be failed to upload, and an upload failure message is sent to the terminal equipment.
Optionally, after sending the upload failure message to the terminal device, the cloud server may also send a prompt message to the terminal device, where the prompt message is used to prompt the user to interrupt transmission of the target file and upload the target file again.
In this embodiment, by performing the locking operation when the data block is written, the problem of data confusion caused by the re-uploading of the data block can be avoided, thereby ensuring the correct uploading of the file.
Fig. 7 is a schematic structural diagram of a document uploading device provided by the present application. The file uploading device of the embodiment may be in the form of software and/or hardware. The file uploading device of the embodiment can be arranged in a cloud server. As shown in fig. 7, the file uploading apparatus 10 of the present embodiment may include: a transceiver module 11, a determination module 12 and a storage module 13.
The receiving and sending module 11 is configured to receive an upload request corresponding to a target file from a terminal device;
the determining module 12 is configured to determine a size of a target block corresponding to the target file according to a current network parameter;
the transceiver module 11 is further configured to send the target block size to the terminal device; the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal device to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the target block size;
the transceiver module 11 is further configured to receive the plurality of data blocks from the terminal device;
the storage module 13 is configured to store the meta information of the target file by using a single-layer index structure.
In a possible implementation manner, the transceiver module 11 is specifically configured to: receiving each of the plurality of data blocks from the terminal device, respectively;
the storage module 13 is specifically configured to:
storing each data block according to the index identification of the target file, and generating and caching the meta information of each data block;
when the plurality of data blocks are all uploaded correctly according to the meta-information of the plurality of data blocks, generating and storing the meta-information of the target file, and clearing the cached meta-information of the plurality of data blocks;
wherein the meta information of the target file comprises an index identifier of the target file.
In a possible implementation manner, the storage module 13 is specifically configured to: aiming at any first data block in the plurality of data blocks, determining an index identifier of the first data block according to the index identifier of the target file;
dividing the first data block into a plurality of data pieces, determining the index identification of each data piece according to the index identification of the first data block, and storing the data pieces according to the index identifications of the data pieces;
and generating and caching the meta-information of the first data block, wherein the meta-information of the first data block comprises the index identification and the fingerprint information of the first data block.
In a possible implementation manner, the storage module 13 is specifically configured to:
calculating the index identification of the target file and the sequence number of the first data block by adopting a first hash operation, and determining the index identification of the first data block;
and calculating the index identification of the first data block and the sequence number of each data slice by adopting a second hash operation, and determining the index identification of each data slice.
Fig. 8 is a schematic structural diagram of another document uploading device provided by the present application. As shown in fig. 8, on the basis of the embodiment shown in fig. 7, the apparatus of this embodiment further includes: an acquisition module 14.
The transceiver module 11 is further configured to: receiving a file query request from the terminal equipment, wherein the file query request is used for requesting to query the content of the target file;
the obtaining module 14 is configured to: acquiring an index identifier of the target file according to the meta information of the target file;
acquiring index identifications of a plurality of data blocks corresponding to the target file according to the index identification of the target file and the first hash operation, and acquiring index identifications of a plurality of data slices corresponding to each data block according to the index identification of each data block and the second hash operation;
and acquiring the content of the target file from a storage space corresponding to the index identifications of the plurality of data pieces.
In a possible implementation manner, the determining module 12 is specifically configured to:
determining whether the upload request is a first upload request for the target file;
and if the uploading request is determined to be the first uploading request aiming at the target file, acquiring the current network parameter, and determining the target block size corresponding to the target file according to the current network parameter.
In a possible implementation manner, the determining module 12 is further configured to:
if the uploading request is determined not to be the first uploading request aiming at the target file, obtaining the size of the historical blocks corresponding to the target file, and adjusting the size of the historical blocks to obtain the size of the target blocks corresponding to the target file.
In a possible implementation manner, the storage module 13 is further configured to:
determining whether the first data block is received for the first time according to the index identification of the first data block;
if the first data block is determined not to be received for the first time, comparing the content of the first data block with the content of a second data block, wherein the second data block is the data block which is received for the first time and has the same index identification with the first data block;
the transceiver module 11 is further configured to: and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.
In a possible implementation manner, the transceiver module 11 is further configured to: and sending prompt information to the terminal equipment, wherein the prompt information is used for prompting a user to interrupt the transmission of the target file and upload the target file again.
In one possible implementation, the current network parameter includes at least one of: the position information of the terminal equipment, the operator information accessed by the terminal equipment, the network transmission rate and the network signal intensity.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided. The electronic device can be used as a cloud server.
Fig. 9 is a block diagram of an electronic device according to a file uploading method in an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 9, the electronic apparatus includes: one or more processors 101, memory 102, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 9 illustrates an example of one processor 101.
The memory 102 is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the file uploading method in the embodiment of the present application (for example, the transceiver module 11, the determination module 12, and the storage module 13 shown in fig. 7, and the acquisition module 14 shown in fig. 8). The processor 101 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 102, that is, implements the file uploading method in the above method embodiments.
The memory 102 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the electronic device of the file upload method, and the like. Further, the memory 102 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 102 may optionally include a memory remotely located from the processor 101, and these remote memories may be connected to the electronic device of the file upload method through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the file uploading method may further include: an input device 103 and an output device 104. The processor 101, the memory 102, the input device 103, and the output device 104 may be connected by a bus or other means, and the bus connection is exemplified in fig. 9.
The input device 103 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus of the file upload method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or other input devices. The output devices 104 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The present application further provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, the technical solution in any of the above method embodiments is implemented. The implementation principle and the technical effect are similar, and the detailed description is omitted here.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (21)
1. A file uploading method comprises the following steps:
receiving an uploading request corresponding to a target file from a terminal device;
determining the size of a target block corresponding to the target file according to the current network parameters, and sending the size of the target block to the terminal equipment; the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal device to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the target block size;
and receiving the data blocks from the terminal equipment, and storing the meta information of the target file by adopting a single-layer index structure.
2. The method of claim 1, wherein the receiving the plurality of data blocks from the terminal device and storing the meta information of the target file in a single-layer index structure comprises:
receiving each data block in the plurality of data blocks from the terminal equipment respectively, storing each data block according to the index identification of the target file, and generating and caching the meta information of each data block;
when the plurality of data blocks are all uploaded correctly according to the meta-information of the plurality of data blocks, generating and storing the meta-information of the target file, and clearing the cached meta-information of the plurality of data blocks;
wherein the meta information of the target file comprises an index identifier of the target file.
3. The method of claim 2, wherein for any first data block in the plurality of data blocks, storing the first data block according to the index identifier of the target file, and generating and caching the meta information of the first data block comprises:
determining the index identifier of the first data block according to the index identifier of the target file;
dividing the first data block into a plurality of data pieces, determining the index identification of each data piece according to the index identification of the first data block, and storing the data pieces according to the index identifications of the data pieces;
and generating and caching the meta-information of the first data block, wherein the meta-information of the first data block comprises the index identification and the fingerprint information of the first data block.
4. The method of claim 3, wherein determining the index identifier of the first data block from the index identifier of the target file comprises:
calculating the index identification of the target file and the sequence number of the first data block by adopting a first hash operation, and determining the index identification of the first data block;
determining the index identifier of each data slice according to the index identifier of the first data block, including:
and calculating the index identification of the first data block and the sequence number of each data slice by adopting a second hash operation, and determining the index identification of each data slice.
5. The method of claim 4, after storing the meta information of the target file in a single-layer index structure, further comprising:
receiving a file query request from the terminal equipment, wherein the file query request is used for requesting to query the content of the target file;
acquiring an index identifier of the target file according to the meta information of the target file;
acquiring index identifications of a plurality of data blocks corresponding to the target file according to the index identification of the target file and the first hash operation, and acquiring index identifications of a plurality of data slices corresponding to each data block according to the index identification of each data block and the second hash operation;
and acquiring the content of the target file from a storage space corresponding to the index identifications of the plurality of data pieces.
6. The method of any one of claims 1 to 5, wherein determining a target chunk size corresponding to the target file according to current network parameters comprises:
determining whether the upload request is a first upload request for the target file;
and if the uploading request is determined to be the first uploading request aiming at the target file, acquiring the current network parameter, and determining the target block size corresponding to the target file according to the current network parameter.
7. The method of claim 6, further comprising:
if the uploading request is determined not to be the first uploading request aiming at the target file, obtaining the size of the historical blocks corresponding to the target file, and adjusting the size of the historical blocks to obtain the size of the target blocks corresponding to the target file.
8. The method according to any one of claims 3 to 5, further comprising, after determining the index identifier of the first data block according to the index identifier of the target file:
determining whether the first data block is received for the first time according to the index identification of the first data block;
if the first data block is determined not to be received for the first time, comparing the content of the first data block with the content of a second data block, wherein the second data block is the data block which is received for the first time and has the same index identification with the first data block;
and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.
9. The method of claim 8, after sending the upload failure message to the terminal device, further comprising:
and sending prompt information to the terminal equipment, wherein the prompt information is used for prompting a user to interrupt the transmission of the target file and upload the target file again.
10. The method of any of claims 1 to 9, the current network parameters comprising at least one of: the position information of the terminal equipment, the operator information accessed by the terminal equipment, the network transmission rate and the network signal intensity.
11. A file uploading apparatus comprising: the receiving and sending module is used for determining the module and the storage module; wherein the content of the first and second substances,
the receiving and sending module is used for receiving an uploading request corresponding to the target file from the terminal equipment;
the determining module is used for determining the size of the target block corresponding to the target file according to the current network parameters;
the transceiver module is further configured to send the size of the target block to the terminal device; the target block size is smaller than or equal to a block size threshold corresponding to the current network parameter, and the target block size is used for enabling the terminal device to divide the target file into a plurality of data blocks, wherein the size of each data block is smaller than or equal to the target block size;
the transceiver module is further configured to receive the plurality of data blocks from the terminal device;
the storage module is used for storing the meta information of the target file by adopting a single-layer index structure.
12. The apparatus of claim 11, wherein the transceiver module is specifically configured to: receiving each of the plurality of data blocks from the terminal device, respectively;
the storage module is specifically configured to:
storing each data block according to the index identification of the target file, and generating and caching the meta information of each data block;
when the plurality of data blocks are all uploaded correctly according to the meta-information of the plurality of data blocks, generating and storing the meta-information of the target file, and clearing the cached meta-information of the plurality of data blocks;
wherein the meta information of the target file comprises an index identifier of the target file.
13. The apparatus of claim 12, wherein the storage module is specifically configured to: aiming at any first data block in the plurality of data blocks, determining an index identifier of the first data block according to the index identifier of the target file;
dividing the first data block into a plurality of data pieces, determining the index identification of each data piece according to the index identification of the first data block, and storing the data pieces according to the index identifications of the data pieces;
and generating and caching the meta-information of the first data block, wherein the meta-information of the first data block comprises the index identification and the fingerprint information of the first data block.
14. The apparatus of claim 13, wherein the storage module is specifically configured to:
calculating the index identification of the target file and the sequence number of the first data block by adopting a first hash operation, and determining the index identification of the first data block;
and calculating the index identification of the first data block and the sequence number of each data slice by adopting a second hash operation, and determining the index identification of each data slice.
15. The apparatus of claim 14, the apparatus further comprising: an acquisition module;
the transceiver module is further configured to: receiving a file query request from the terminal equipment, wherein the file query request is used for requesting to query the content of the target file;
the acquisition module is configured to: acquiring an index identifier of the target file according to the meta information of the target file;
acquiring index identifications of a plurality of data blocks corresponding to the target file according to the index identification of the target file and the first hash operation, and acquiring index identifications of a plurality of data slices corresponding to each data block according to the index identification of each data block and the second hash operation;
and acquiring the content of the target file from a storage space corresponding to the index identifications of the plurality of data pieces.
16. The apparatus according to any one of claims 11 to 15, wherein the determining means is specifically configured to:
determining whether the upload request is a first upload request for the target file;
and if the uploading request is determined to be the first uploading request aiming at the target file, acquiring the current network parameter, and determining the target block size corresponding to the target file according to the current network parameter.
17. The apparatus of claim 16, the determination module further to:
if the uploading request is determined not to be the first uploading request aiming at the target file, obtaining the size of the historical blocks corresponding to the target file, and adjusting the size of the historical blocks to obtain the size of the target blocks corresponding to the target file.
18. The apparatus of any of claims 13 to 15, the storage module further to:
determining whether the first data block is received for the first time according to the index identification of the first data block;
if the first data block is determined not to be received for the first time, comparing the content of the first data block with the content of a second data block, wherein the second data block is the data block which is received for the first time and has the same index identification with the first data block;
the transceiver module is further configured to: and when the comparison result is inconsistent, sending an uploading failure message to the terminal equipment.
19. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.
20. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1 to 10.
21. A computer program product comprising computer instructions which, when executed by a processor, implement the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011406833.XA CN112583904B (en) | 2020-12-04 | 2020-12-04 | File uploading method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011406833.XA CN112583904B (en) | 2020-12-04 | 2020-12-04 | File uploading method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112583904A true CN112583904A (en) | 2021-03-30 |
| CN112583904B CN112583904B (en) | 2023-05-19 |
Family
ID=75127228
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011406833.XA Active CN112583904B (en) | 2020-12-04 | 2020-12-04 | File uploading method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112583904B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039735A (en) * | 2021-11-19 | 2022-02-11 | 广州金山移动科技有限公司 | Method and device for transmitting data between devices |
| CN114584556A (en) * | 2022-03-14 | 2022-06-03 | 中国工商银行股份有限公司 | File transmission method and device |
| CN115065676A (en) * | 2022-05-17 | 2022-09-16 | 深圳市绿联科技股份有限公司 | Method and device for uploading files and electronic equipment |
| CN117459525A (en) * | 2023-12-20 | 2024-01-26 | 宁德时代新能源科技股份有限公司 | File uploading method, device, equipment and medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100094817A1 (en) * | 2008-10-14 | 2010-04-15 | Israel Zvi Ben-Shaul | Storage-network de-duplication |
| CN102163199A (en) * | 2010-02-24 | 2011-08-24 | 富士通株式会社 | Index construction method and device thereof and query method |
| CN102332027A (en) * | 2011-10-15 | 2012-01-25 | 西安交通大学 | Mass non-independent small file associated storage method based on Hadoop |
| US8370315B1 (en) * | 2010-05-28 | 2013-02-05 | Symantec Corporation | System and method for high performance deduplication indexing |
| US20140250155A1 (en) * | 2011-11-17 | 2014-09-04 | Huawei Technologies Co., Ltd. | Metadata storage and management method for cluster file system |
| CN104091129A (en) * | 2014-06-26 | 2014-10-08 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| US8935446B1 (en) * | 2013-09-26 | 2015-01-13 | Emc Corporation | Indexing architecture for deduplicated cache system of a storage system |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| US20150178007A1 (en) * | 2013-03-15 | 2015-06-25 | Cloudifyd, Inc. | Method and system for integrated cloud storage management |
| CN105138632A (en) * | 2015-08-20 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | Organization and management method for file data and file management server |
| US9430156B1 (en) * | 2014-06-12 | 2016-08-30 | Emc Corporation | Method to increase random I/O performance with low memory overheads |
| US20170344573A1 (en) * | 2016-05-26 | 2017-11-30 | Research & Business Foundation Sungkyunkwan University | Data discard method for journaling file system and memory management apparatus thereof |
| CN107872489A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file section method for uploading, device and cloud storage system |
| CN110569213A (en) * | 2018-05-18 | 2019-12-13 | 北京果仁宝软件技术有限责任公司 | File access method, device and equipment |
| CN111327694A (en) * | 2020-02-10 | 2020-06-23 | 北京达佳互联信息技术有限公司 | File uploading method and device, storage medium and electronic equipment |
-
2020
- 2020-12-04 CN CN202011406833.XA patent/CN112583904B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100094817A1 (en) * | 2008-10-14 | 2010-04-15 | Israel Zvi Ben-Shaul | Storage-network de-duplication |
| CN102163199A (en) * | 2010-02-24 | 2011-08-24 | 富士通株式会社 | Index construction method and device thereof and query method |
| US8370315B1 (en) * | 2010-05-28 | 2013-02-05 | Symantec Corporation | System and method for high performance deduplication indexing |
| CN102332027A (en) * | 2011-10-15 | 2012-01-25 | 西安交通大学 | Mass non-independent small file associated storage method based on Hadoop |
| US20140250155A1 (en) * | 2011-11-17 | 2014-09-04 | Huawei Technologies Co., Ltd. | Metadata storage and management method for cluster file system |
| US20150178007A1 (en) * | 2013-03-15 | 2015-06-25 | Cloudifyd, Inc. | Method and system for integrated cloud storage management |
| US8935446B1 (en) * | 2013-09-26 | 2015-01-13 | Emc Corporation | Indexing architecture for deduplicated cache system of a storage system |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| US9430156B1 (en) * | 2014-06-12 | 2016-08-30 | Emc Corporation | Method to increase random I/O performance with low memory overheads |
| CN104091129A (en) * | 2014-06-26 | 2014-10-08 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN105138632A (en) * | 2015-08-20 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | Organization and management method for file data and file management server |
| US20170344573A1 (en) * | 2016-05-26 | 2017-11-30 | Research & Business Foundation Sungkyunkwan University | Data discard method for journaling file system and memory management apparatus thereof |
| CN107872489A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file section method for uploading, device and cloud storage system |
| CN110569213A (en) * | 2018-05-18 | 2019-12-13 | 北京果仁宝软件技术有限责任公司 | File access method, device and equipment |
| CN111327694A (en) * | 2020-02-10 | 2020-06-23 | 北京达佳互联信息技术有限公司 | File uploading method and device, storage medium and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| 付松龄;廖湘科;黄辰林;王蕾;李姗姗;: "FlatLFS:一种面向海量小文件处理优化的轻量级文件系统" * |
| 孙竞;余宏亮;郑纬民;: "支持分布式存储删冗的相似文件元数据集合索引" * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039735A (en) * | 2021-11-19 | 2022-02-11 | 广州金山移动科技有限公司 | Method and device for transmitting data between devices |
| CN114584556A (en) * | 2022-03-14 | 2022-06-03 | 中国工商银行股份有限公司 | File transmission method and device |
| CN115065676A (en) * | 2022-05-17 | 2022-09-16 | 深圳市绿联科技股份有限公司 | Method and device for uploading files and electronic equipment |
| CN117459525A (en) * | 2023-12-20 | 2024-01-26 | 宁德时代新能源科技股份有限公司 | File uploading method, device, equipment and medium |
| CN117459525B (en) * | 2023-12-20 | 2024-04-12 | 宁德时代新能源科技股份有限公司 | File uploading method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112583904B (en) | 2023-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112583904B (en) | File uploading method, device, equipment and storage medium | |
| EP3602305B1 (en) | Rule-based modifications in a data storage appliance monitor | |
| US9417811B2 (en) | Efficient inline data de-duplication on a storage system | |
| US20200159661A1 (en) | Cache Coherence for File System Interfaces | |
| US11296940B2 (en) | Centralized configuration data in a distributed file system | |
| KR20150132859A (en) | Automatic tuning of virtual data center resource utilization policies | |
| US10305985B1 (en) | Defining new properties using expressions in API calls | |
| US11347647B2 (en) | Adaptive cache commit delay for write aggregation | |
| CN109918191B (en) | Method and device for preventing frequency of service request | |
| US11263174B2 (en) | Reducing resource consumption in container image management | |
| US10802922B2 (en) | Accelerated deduplication block replication | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| JP7210642B2 (en) | File directory traversal method, device, facility, medium, and program | |
| US11500577B2 (en) | Method, electronic device, and computer program product for data processing | |
| CN112565356A (en) | Data storage method and device and electronic equipment | |
| CN110619008A (en) | Database query method and device, electronic equipment and storage medium | |
| US9223822B1 (en) | Techniques for performing indication management | |
| US11176097B2 (en) | Accelerated deduplication block replication | |
| US8447729B1 (en) | Techniques for performing indication management | |
| CN113778973A (en) | Data storage method and device | |
| CN113726885A (en) | Method and device for adjusting flow quota | |
| US9191267B2 (en) | Device management for determining the effects of management actions | |
| KR101754713B1 (en) | Asymmetric distributed file system, apparatus and method for distribution of computation | |
| CN112925482B (en) | Data processing method, device, system, electronic equipment and computer storage medium | |
| CN113051121B (en) | Log information retrieval method, device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |