CN114827130B - File uploading method and device - Google Patents
File uploading method and device Download PDFInfo
- Publication number
- CN114827130B CN114827130B CN202210454903.1A CN202210454903A CN114827130B CN 114827130 B CN114827130 B CN 114827130B CN 202210454903 A CN202210454903 A CN 202210454903A CN 114827130 B CN114827130 B CN 114827130B
- Authority
- CN
- China
- Prior art keywords
- file
- uploading
- uploaded
- test
- size
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012360 testing method Methods 0.000 claims abstract description 122
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012545 processing Methods 0.000 abstract description 11
- 238000005516 engineering process Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 239000000872 buffer Substances 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for uploading files. Receiving a file to be uploaded sent by a client; determining an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a preset threshold value, wherein the preset threshold value is used for determining test results of multiple uploading tests on a plurality of test files with different sizes according to a plurality of uploading modes; and uploading the file to be uploaded to a storage server according to the uploading mode. According to the embodiment of the invention, the problems that the size of the file to be uploaded sent by the client is not fixed, if the size of the file to be uploaded received by the server is too large, the storage space of the server is occupied too much to influence the normal operation of the server due to the fact that the cache is carried out in the memory, and if the size of the file to be uploaded received by the server is smaller, the uploading efficiency of the file to be uploaded is seriously reduced due to the fact that the cache is carried out in the disk are solved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for uploading files.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the rapid development of technology, more and more businesses need to be uploaded to a cloud storage server for backup. If a plurality of clients are connected with the storage server, corresponding network bandwidth needs to be allocated to each client, and when the clients do not upload files, the network bandwidth allocated to the clients is in an idle state, and the idle network bandwidth cannot be used by other clients, so that the waste of the network bandwidth is caused. The client side needs to send the file to be uploaded to a server side connected with the cloud storage server, and the file is uploaded to the storage server by the server side. In the prior art, when uploading a file, a server needs to buffer the received file to be uploaded in a memory or a disk, and then send the file to a cloud storage server. Because the size of the file to be uploaded sent by the client is not fixed, if the size of the file to be uploaded received by the server is too large, the server caches the received too large file to be uploaded in the memory, which causes the storage space of the server to be occupied too much, and influences the normal operation of the server, and if the size of the file to be uploaded received by the server is smaller, the server caches the received smaller file to be uploaded in the disk, which seriously reduces the uploading efficiency of the file to be uploaded.
The method for uploading the file is needed at present, so that the problem that in the prior art, as the size of the file to be uploaded received by the server is not fixed, if the file to be uploaded received by the server is too large, the server caches the received too large file to be uploaded in the memory, so that the normal operation of the server is affected, and if the file to be uploaded received by the server is smaller, the server caches the received smaller file to be uploaded in the disk, so that the uploading efficiency of the file to be uploaded is seriously reduced is solved.
Disclosure of Invention
In order to solve the problems in the prior art, the embodiment of the invention provides a file uploading method and a file uploading device, which realize that a file uploading mode is determined according to the size of a file to be uploaded, and solve the problems in the prior art that the size of the file to be uploaded received by a server is not fixed, if the size of the file to be uploaded received by the server is too large, the server caches the received too large file to be uploaded in a memory, so that the storage space of the server is occupied too much, the normal operation of the server is affected, and if the size of the file to be uploaded received by the server is smaller, the uploading efficiency of the file to be uploaded is seriously reduced by caching the received smaller file to be uploaded in a disk.
In order to solve the technical problems, the specific technical scheme of the invention is as follows:
in one aspect, an embodiment of the present invention provides a method for uploading a file, where the method includes:
receiving a file to be uploaded sent by a client;
determining an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a preset threshold value, wherein the preset threshold value is used for determining test results of multiple uploading tests on a plurality of test files with different sizes according to a plurality of uploading modes;
and uploading the file to be uploaded to a storage server according to the uploading mode.
Further, the uploading mode comprises a file stream uploading mode and a file block uploading mode;
the file stream uploading mode comprises the steps of caching the file to be uploaded in a file receiving cache area in a memory, and uploading the file to be uploaded cached in the file receiving cache area to the storage server;
the file block uploading mode comprises the steps of generating a file block according to the file to be uploaded, caching the file block in a file caching unit in a disk, and uploading the file block cached in the file caching unit to the storage server.
Further, the file block uploading mode also comprises,
splitting the file block into a plurality of fragmented files, wherein the size of the fragmented files is smaller than or equal to the size of the file uploaded at a time;
and uploading the segmented files to the storage server in sequence according to the splitting sequence of the segmented files.
Further, the plurality of adjacent fragmented files have partially identical content, and the content identity ratio between the adjacent fragmented files is at least 50%.
Further, the step of determining the predetermined threshold comprises,
step 1: determining an initial threshold according to the size of the system memory;
step 2: determining the size of a first test file and the size of a second test file according to the initial threshold value so as to upload the first test file to the storage server in the file stream uploading mode, upload the second test file to the storage server in the file block uploading mode, and respectively record uploading time of the first test file and the second test file;
step 3: adjusting the initial threshold according to a preset step length, and repeating the step 2 to respectively obtain uploading time of the first test file and the second test file corresponding to each initial threshold;
Step 4: and determining the preset threshold according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold.
Further, determining the size of the first test file and the size of the second test file based on the initial threshold includes,
calculating the difference between the initial threshold value and a preset adjustment amount to obtain the size of the first test file, wherein the preset adjustment amount is a positive number;
and calculating the sum of the initial threshold value and the preset adjustment quantity to obtain the size of the second test file.
Further, determining the predetermined threshold value further includes, based on each initial threshold value and the upload time of the first test file and the second test file corresponding to each initial threshold value,
respectively drawing uploading curves of the first test file and the second test file by taking the initial threshold value as an abscissa and the uploading time as an ordinate;
determining an intersection point of uploading curves of the first test file and the second test file;
and taking the initial value of the threshold corresponding to the intersection point as the preset threshold.
On the other hand, the embodiment of the invention also provides a file uploading device, which comprises:
The file to be uploaded receiving unit is used for receiving the file to be uploaded sent by the client;
an uploading mode determining unit, configured to determine an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a predetermined threshold, where the predetermined threshold determines test results of multiple uploading tests on multiple test files with different sizes according to multiple uploading modes;
and the file to be uploaded uploading unit is used for uploading the file to be uploaded to a storage server according to the uploading mode.
In another aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the method when executing the computer program.
In another aspect, embodiments of the present invention further provide a computer readable storage medium storing a computer program that when executed by a processor implements the above method.
Finally, an embodiment of the invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.
In the embodiment of the invention, after receiving the file to be uploaded sent by the client, the uploading mode of the file to be uploaded is determined according to the size of the file to be uploaded and a preset threshold value, and then the file to be uploaded is uploaded to the storage server according to the determined uploading mode. The method comprises the steps of determining a preset threshold according to test results of multiple uploading tests on a plurality of test files with different sizes in a plurality of uploading modes, representing performance conditions of a server and a storage server in a plurality of uploading modes, determining the uploading mode according to the preset threshold and the sizes of the files to be uploaded, uploading the files to be uploaded, and utilizing the performances of the server and the storage server to the greatest extent, so that uploading efficiency of the files to be uploaded is improved, and the probability of faults of the server and the storage server is reduced. Compared with the scheme of uploading the file to be uploaded by adopting the same uploading mode in the prior art, the method solves the problems that if the file to be uploaded is received by the server side and is overlarge, the server side caches the received overlarge file to be uploaded in the memory, so that the storage space of the server side is occupied too much to influence the normal operation of the server side, and if the file to be uploaded is received by the server side to be smaller, the server side caches the received smaller file to be uploaded in the disk to seriously reduce the uploading efficiency of the file to be uploaded.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a schematic diagram of an implementation system of a file uploading method according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for uploading files according to an embodiment of the present invention;
FIG. 3 is a process of determining a predetermined threshold in an embodiment of the present invention;
FIG. 4 is a process of determining a size of a first test file and a size of a second test file according to an initial threshold in an embodiment of the present invention;
FIG. 5 is a process of determining a predetermined threshold according to each initial threshold and uploading time of a first test file and a second test file corresponding to each initial threshold in an embodiment of the present invention
FIG. 6 is a schematic diagram of a file uploading device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Description of the drawings:
101. a client;
102. a server;
103. a storage server;
601. a file receiving unit to be uploaded;
602. an uploading mode determining unit;
603. a file uploading unit to be uploaded;
702. a computer device;
704. a processing device;
706. storing the resource;
708. a driving mechanism;
710. an input/output module;
712. an input device;
714. an output device;
716. a presentation device;
718. a graphical user interface;
720. a network interface;
722. a communication link;
724. a communication bus.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present invention and their descriptions herein are for the purpose of explaining the present invention, but are not to be construed as limiting the invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or device.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
Fig. 1 is a schematic diagram of an implementation system of a file uploading method according to an embodiment of the present invention, which may include: the client 101, the server 102, and the storage server 103 communicate between the client 101 and the server 102, and between the server 102 and the storage server 103 via a network, which may include a local area network (Local Area Network, abbreviated as LAN), a wide area network (Wide Area Network, abbreviated as WAN), the internet, or a combination thereof, and is connected to a website, user equipment (e.g., a computing device), and a back-end system. When the client 101 needs to upload a file to the storage server 103, the client 101 sends the file to be uploaded to the server 102, and the server 102 uploads the file sent by the client 101 to the storage server 103. The server 102 is provided with an upload processing system for files to be uploaded, and the storage server 103 is provided with a database for storing files uploaded by the client 101, alternatively, the storage server 103 may be a node (not shown in the figure) for cloud storage, or each storage server 103 may be a separate database including a plurality of computers interconnected by a network and operating as a distributed storage system.
In an alternative embodiment, the client 101 may include, but is not limited to, a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an augmented Reality (AR, augmented Reality)/Virtual Reality (VR) device, a smart wearable device, and the like. Alternatively, the operating system running on the electronic device may include, but is not limited to, an android system, an IOS system, linux, windows, and the like.
In addition, it should be noted that, fig. 1 is only an application environment provided by the present disclosure, and in practical application, a plurality of clients 101 may also be included, which is not limited in this specification.
At present, after the server 102 receives the file to be uploaded sent by the client 101, the received file to be uploaded needs to be cached in a memory or a disk, and then is uploaded to the storage server 103, because the size of the file to be uploaded sent by the client 101 is not fixed, if the size of the file to be uploaded received by the server 102 is too large, caching in the memory will cause that the storage space of the server 102 is occupied too much, which affects the normal operation of the server 102, and if the size of the file to be uploaded received by the server 102 is smaller, caching in the disk will seriously reduce the uploading efficiency of the file to be uploaded.
Aiming at the problems existing in the prior art, the embodiment of the invention provides a file uploading method which can be applied to a server 102, and the method for determining the uploading mode of the file to be uploaded according to the size of the file to be uploaded is realized, and the file to be uploaded is uploaded to a storage server through the determined uploading mode. Fig. 2 is a flow chart of a file uploading method according to an embodiment of the invention. The process of uploading a file to be uploaded from a server to a terminal to a storage server is described in this figure, but may include more or fewer operational steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one way of performing the order of steps and does not represent a unique order of execution. When a system or apparatus product in practice is executed, it may be executed sequentially or in parallel according to the method shown in the embodiments or the drawings. As shown in fig. 2, the method may be executed by the server, and the method may include:
step 201: receiving a file to be uploaded sent by a client;
step 202: determining an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a preset threshold value, wherein the preset threshold value is used for determining test results of multiple uploading tests on a plurality of test files with different sizes according to a plurality of uploading modes;
Step 203: and uploading the file to be uploaded to a storage server according to the uploading mode.
According to the method, after the file to be uploaded sent by the client is received, the uploading mode of the file to be uploaded is determined according to the size of the file to be uploaded and the preset threshold value, and then the file to be uploaded is uploaded to the storage server according to the determined uploading mode. Compared with the scheme of uploading files to be uploaded by adopting the same uploading mode in the prior art, the method solves the problems that the size of the files to be uploaded sent by a client is not fixed, when the files to be uploaded are received by a server to be oversized, the storage space of the server is occupied too much to influence the normal operation of the server, and if the files to be uploaded received by the server are smaller, the uploading efficiency of the files to be uploaded is seriously reduced by caching in a disk.
In the embodiment of the present invention, the type of the file to be uploaded sent by the client may be video, document, system version, etc., and the embodiment of the present invention is not limited. The client may send the complete file to be uploaded to the server, or split the file to be uploaded into multiple fragmented files, and then send each fragmented file to the server. The size of the file to be uploaded, which is sent to the server by the client each time, is determined by the client according to the service condition of the client. After the server receives the file to be uploaded or the fragmented file of the file to be uploaded, determining an uploading mode according to the size of the received file to be uploaded or the fragmented file, wherein the uploading mode can be an uploading mode agreed by the server in advance with the storage server, for example, the storage server provides an API interface of the uploading mode for the server, and the server uploads the file to be uploaded to the storage server by calling the API interface of the uploading mode corresponding to the size of the received file to be uploaded, and finally the storage server stores the file to be uploaded.
The predetermined threshold is determined according to test results of multiple uploading tests on a plurality of test files with different sizes in a plurality of uploading modes, and it can be understood that the size of the predetermined threshold can represent performance conditions of the server and the storage server under a plurality of uploading modes, the uploading mode is determined according to the predetermined threshold and the size of the file to be uploaded, and the file to be uploaded is uploaded, so that the performance of the server and the storage server can be utilized to the greatest extent, the uploading efficiency of the file to be uploaded is improved, and the probability of faults of the server and the storage server is reduced.
According to one embodiment of the invention, the uploading mode comprises a file stream uploading mode and a file block uploading mode;
the file stream uploading mode comprises the steps of caching the file to be uploaded in a file receiving cache area in a memory, and uploading the file to be uploaded cached in the file receiving cache area to the storage server;
the file block uploading mode comprises the steps of generating a file block according to the file to be uploaded, caching the file block in a file caching unit in a disk, and uploading the file block cached in the file caching unit to the storage server.
The file type sent by the client to the server may be a fragmented file or a complete file, which is understood that, in a file stream uploading manner, the server caches the file in a file receiving buffer area in the memory, optionally, the size of the file receiving buffer area in the memory may be allocated in advance, and the size of the file receiving buffer area may be determined according to a predetermined threshold. When a plurality of clients correspond to a server, the file uploading method shown in fig. 2 can be executed by different file uploading threads of the server, at this time, one thread can correspond to a file receiving buffer zone, each thread buffers the file to be uploaded in a respective file receiving buffer zone, reads the buffered file to be uploaded from the respective file buffer zone and sends the file to a storage server, so that ordered uploading of the file to be uploaded is ensured.
And under the file block uploading mode, the server caches the file in a file cache unit in the disk. Since the size of the file to be uploaded is not fixed, the size of the file cache unit may not be set in advance. When a plurality of clients correspond to a server, the file uploading method shown in fig. 2 can be executed by different file uploading threads of the server, at this time, one thread can correspond to a file caching unit, each thread caches the file to be uploaded in a respective file caching unit, and the cached file to be uploaded is read from the respective file caching unit and sent to the storage server, so that ordered uploading of the file to be uploaded is ensured.
In the embodiment of the invention, when the size of the file to be uploaded is smaller than the preset threshold, the uploading mode of the file to be uploaded can be determined to be a file stream uploading mode, and when the size of the file to be uploaded is larger than or equal to the preset threshold, the uploading mode of the file to be uploaded can be determined to be a file block uploading mode. It can be understood that, since the cache and reading efficiency of the file in the memory is far higher than that of the disk, the file to be uploaded with the file size smaller than the predetermined threshold is uploaded by a file stream uploading mode. However, if the size of the file to be uploaded is too large, if the file stream uploading mode is still adopted for uploading, the memory of the server side may be exhausted, and the normal operation of the server side is affected. By the method provided by the embodiment of the invention, the file stream uploading mode or the file block uploading mode is determined according to the size of the file to be uploaded and the preset threshold value, so that the uploading efficiency of the file to be uploaded is improved, and the normal operation of the server side is ensured.
In the embodiment of the invention, when uploading is performed by adopting a file block uploading mode, the uploaded file block may be oversized, if an error occurs in the uploading process, the file block can only be uploaded again, so that the retransmission time is long, and the uploading efficiency of the file to be uploaded is reduced. In view of the foregoing, according to one embodiment of the present invention, the file block uploading manner further includes,
Splitting the file block into a plurality of fragmented files, wherein the size of the fragmented files is smaller than or equal to the size of the file uploaded at a time;
and uploading the segmented files to the storage server in sequence according to the splitting sequence of the segmented files.
In the embodiment of the invention, when the size of a file to be uploaded is larger than or equal to a preset threshold value, a server generates a file block from the file to be uploaded, and after the file block is cached in a file caching unit in a disk, the file block is split into a plurality of fragmented files according to an uploading rule agreed with a storage server, wherein the uploading rule comprises the file size of single uploading; and then uploading the segmented files to the storage server in turn according to the splitting sequence of the segmented files.
It can be understood that when a file block is split into a plurality of fragmented files and each fragmented file is uploaded, if a certain fragmented file fails to be uploaded, the server side only needs to upload the fragmented file again, and the file block does not need to be retransmitted any more, so that the retransmission time is reduced, and the uploading efficiency of the file to be uploaded is improved.
In the embodiment of the invention, when uploading is performed in a file block uploading mode, the server side may cause packet loss of the fragmented file uploaded to the storage server due to the network link, and even if the packet loss is small, the server side needs to re-upload the fragmented file in order to ensure the integrity of the uploaded file, thereby increasing the workload of the server side. In view of the above, according to one embodiment of the present invention, a plurality of adjacent fragmented files have partially identical contents, and the content identical rate between the adjacent fragmented files is at least 50%.
It can be understood that the plurality of adjacent fragmented files have partially identical contents, when a packet loss occurs in the uploading process of a certain fragmented file, the storage server can still obtain the content of the packet loss from the plurality of adjacent fragmented files of the packet loss fragmented file, thereby ensuring the integrity of the file to be uploaded. When the server splits the file block into a plurality of fragmented files, the content of the file block needs to be marked, so that when a certain fragmented file loses a packet, the storage server can obtain the lost content from a plurality of adjacent fragmented files of the lost fragmented file according to the mark of the content. The content of the file block may be marked by a method of marking a file in the prior art, which is not described in detail in this specification.
Preferably, the method for having partially identical content in a plurality of adjacent fragmented files may include the steps of:
step 1: dividing the size of the file uploaded once according to a preset factor to obtain the size of the minimum content unit;
step 2: marking the contents of the file blocks in sequence according to the size of the minimum content unit to obtain a plurality of marked minimum content units;
Step 3: and splitting a plurality of file blocks with marks into a plurality of segmented files by adopting a sliding window method according to the file size of single uploading and a preset step length, wherein the preset step length is equal to an integral multiple of the size of the minimum content unit, and each segmented file comprises a plurality of minimum content units with marks so that the storage server integrates the minimum content units in the segmented file according to the received marks of the minimum content units in the segmented file to obtain the uploaded file to be uploaded.
In the embodiment of the invention, the predetermined factor can be determined according to the size of the fragmented file, the size of the predetermined factor directly determines the size of the minimum content unit, and then the file block is marked by utilizing the size of the minimum content unit, which can be understood that the smaller the minimum content unit is, the higher the refinement degree of the file block content is, correspondingly, the higher the refinement degree of the content split into a plurality of fragmented files is, if a certain fragmented file loses a packet in the uploading process, the lost packet content can be more finely positioned, so that the lost packet content is obtained in a plurality of adjacent fragmented files of the lost packet fragmented file according to the marking of the content.
In the embodiment of the invention, the preset step length can be determined according to the file uploading cost, and when the packet loss occurs in the uploading process of a certain fragmented file, the packet loss content is obtained in a plurality of adjacent fragmented files of the packet loss fragmented file according to the mark of the content, so that the success rate of the complete uploading file is higher, and the file uploading cost is correspondingly higher.
For example, if the size of the file in a single upload is 100KB and the predetermined factor is 5, the size of the minimum content unit is 20KB, and the contents of the file block A are marked sequentially according to the size of the minimum content unit 20KB to obtain a plurality of marked minimum content units A 1 、A 2 、…、A n ，A 1 ～A n If the predetermined step length is 2 times of the minimum content unit 20KB, the predetermined step length is 40KB, and a sliding window method is adopted to split a plurality of file blocks A with marks into a plurality of fragmented files, wherein the obtained fragmented files are as follows: a is that 1 +A 2 +A 3 +A 4 +A 5 、A 3 +A 4 +A 5 +A 6 +A 7 、A 5 +A 6 +A 7 +A 8 +A 9 …, where "+" indicates that the two minimum unit contents are immediately adjacent, then the fragmented files A are each 1 +A 2 +A 3 +A 4 +A 5 、A 3 +A 4 +A 5 +A 6 +A 7 、A 5 +A 6 +A 7 +A 8 +A 9 …, uploading to a storage server, if the file A is fragmented 3 +A 4 +A 5 +A 6 +A 7 Minimum content unit a of (2) 5 Packet loss occurs, then the packet can be found in the fragment file A adjacent to the fragment file 1 +A 2 +A 3 +A 4 +A 5 、A 5 +A 6 +A 7 +A 8 +A 9 Is the smallest content unit A 5 Avoiding re-slicing file a 3 +A 4 +A 5 +A 6 +A 7 Uploading.
According to one embodiment of the present invention, as shown in fig. 3, the process of determining the predetermined threshold includes,
step 301: determining an initial threshold according to the size of the system memory;
step 302: determining the size of a first test file and the size of a second test file according to the initial threshold value so as to upload the first test file to the storage server in the file stream uploading mode, upload the second test file to the storage server in the file block uploading mode, and respectively record uploading time of the first test file and the second test file;
step 303: adjusting the initial threshold according to a preset step length, and repeating the step 302 to respectively obtain the uploading time of the first test file and the second test file corresponding to each initial threshold;
step 304: and determining the preset threshold according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold.
In the embodiment of the invention, an initial threshold is firstly determined according to the memory size of a server system, the unit of the initial threshold is the size of a file, and then the size of a first test file and the size of a second test file are determined according to the initial threshold, so that the first test file is conveniently uploaded to a storage server in a file stream uploading mode, and the second test file is conveniently uploaded to the storage server in a file block uploading mode. Preferably, according to one embodiment of the present invention, as shown in fig. 4, the step 302 of determining the size of the first test file and the size of the second test file according to the initial threshold comprises,
Step 401: calculating the difference between the initial threshold value and a preset adjustment amount to obtain the size of the first test file, wherein the preset adjustment amount is a positive number;
step 402: and calculating the sum of the initial threshold value and the preset adjustment quantity to obtain the size of the second test file.
And in the uploading process of the first test file and the second test file, recording the uploading time of the first test file and the second test file. And then, adjusting the initial threshold according to a preset step length to obtain an adjusted initial threshold, and repeating the step 302 to respectively obtain the uploading time of the first test file and the second test file corresponding to each initial threshold. And finally, determining a preset threshold according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold.
By the method shown in fig. 3 and fig. 4, the determined initial threshold value may represent uploading time when the server and the storage server upload files according to test files with different sizes and different uploading modes, the uploading time may represent performance of the server and the storage server, and the optimal uploading mode of the files to be uploaded can be determined by the initial threshold value obtained by the method shown in fig. 3 and fig. 4 and the size of the files to be uploaded received by the server, so that performance utilization of the server and the storage server can be maximized.
In order to improve the accuracy of calculating the initial threshold according to an embodiment of the present invention, as shown in fig. 5, step 304 further includes, according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold, determining the predetermined threshold,
step 501: respectively drawing uploading curves of the first test file and the second test file by taking the initial threshold value as an abscissa and the uploading time as an ordinate;
step 502: determining an intersection point of uploading curves of the first test file and the second test file;
step 503: and taking the initial value of the threshold corresponding to the intersection point as the preset threshold.
When uploading the file to be uploaded by the file stream uploading method of the embodiment of the invention, the server stores the received file to be uploaded in the file cache region of the memory in the file stream uploading method, the speed of caching and reading the file to be uploaded from the memory is higher, and when the cached file to be uploaded is smaller, the uploading speed of the file to be uploaded is higher, but as the cached file to be uploaded is gradually increased, the memory occupancy rate of the server is gradually increased, so that the response speed of the server is reduced, and the uploading speed of the file to be uploaded is gradually reduced, therefore, the uploading curve of the first test file uploaded by the file stream uploading method is a decreasing curve.
When uploading the file to be uploaded by the file block uploading mode of the embodiment of the invention, the server stores the received file to be uploaded in the file caching unit of the disk in the file block uploading mode, and caches the file to be uploaded from the disk, and the reading speed is low, but as the cached file to be uploaded gradually increases, the file uploading speed gradually increases due to the gradual increase of the size of the file to be uploaded once, and the uploading curve of the second test file uploaded by the file block uploading mode is an incremental curve.
Therefore, an intersection point of the uploading curve of the first test file and the uploading curve of the second test file is determined, the ordinate uploading time corresponding to the intersection point is the uploading time with the optimal uploading mode of the file stream and the file block, and the abscissa threshold value is the optimal preset threshold value.
The embodiment of the invention also provides a file uploading device, as shown in figure 6, comprising,
the file to be uploaded receiving unit 601 is configured to receive a file to be uploaded sent by a client;
an upload mode determining unit 602, configured to determine an upload mode of the file to be uploaded according to the size of the file to be uploaded and a predetermined threshold, where the predetermined threshold determines test results of multiple upload tests performed on multiple test files with different sizes according to multiple upload modes;
And the file to be uploaded uploading unit 603 is configured to upload the file to be uploaded to a storage server according to the uploading manner.
Since the principle of the device for solving the problem is similar to that of the method, the implementation of the device can be referred to the implementation of the method, and the repetition is omitted.
Fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention, where the apparatus in the present invention may be the computer device in the embodiment, and the method of the present invention is performed. The computer device 702 may include one or more processing devices 704, such as one or more Central Processing Units (CPUs), each of which may implement one or more hardware threads. The computer device 702 may also include any storage resources 706 for storing any kind of information, such as code, settings, data, etc. For example, and without limitation, storage resources 706 may include any one or more of the following combinations: any type of RAM, any type of ROM, flash memory devices, hard disks, optical disks, etc. More generally, any storage resource may store information using any technology. Further, any storage resource may provide volatile or non-volatile retention of information. Further, any storage resources may represent fixed or removable components of computer device 702. In one case, the computer device 702 can perform any of the operations of the associated instructions when the processing device 704 executes the associated instructions stored in any storage resource or combination of storage resources. The computer device 702 also includes one or more drive mechanisms 708, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like, for interacting with any storage resources.
The computer device 702 may also include an input/output module 710 (I/O) for receiving various inputs (via an input device 712) and for providing various outputs (via an output device 714). One particular output mechanism may include a presentation device 716 and an associated Graphical User Interface (GUI) 718. In other embodiments, input/output module 710 (I/O), input device 712, and output device 714 may not be included as just one computer device in a network. The computer device 702 can also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the above-described components together.
Communication link 722 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the internet), a point-to-point connection, etc., or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-described method.
Embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements the above method.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (8)
1. A file uploading method is characterized in that the method comprises the following steps of,
receiving a file to be uploaded sent by a client;
determining an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a preset threshold value, wherein the preset threshold value is used for determining test results of multiple uploading tests on a plurality of test files with different sizes according to a plurality of uploading modes;
uploading the file to be uploaded to a storage server according to the uploading mode;
the uploading mode comprises a file stream uploading mode and a file block uploading mode;
the file stream uploading mode comprises the steps of caching the file to be uploaded in a file receiving cache area in a memory, and uploading the file to be uploaded cached in the file receiving cache area to the storage server;
the file block uploading mode comprises the steps of generating a file block according to the file to be uploaded, caching the file block in a file caching unit in a disk, and uploading the file block cached in the file caching unit to the storage server;
the step of determining the predetermined threshold value comprises,
step 1: determining an initial threshold according to the size of the system memory;
Step 2: determining the size of a first test file and the size of a second test file according to the initial threshold value so as to upload the first test file to the storage server in the file stream uploading mode, upload the second test file to the storage server in the file block uploading mode, and respectively record uploading time of the first test file and the second test file;
step 3: adjusting the initial threshold according to a preset step length, and repeating the step 2 to respectively obtain uploading time of the first test file and the second test file corresponding to each initial threshold;
step 4: determining the preset threshold according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold;
determining the size of the first test file and the size of the second test file based on the initial threshold includes,
calculating the difference between the initial threshold value and a preset adjustment amount to obtain the size of the first test file, wherein the preset adjustment amount is a positive number;
and calculating the sum of the initial threshold value and the preset adjustment quantity to obtain the size of the second test file.
2. The method for uploading files according to claim 1, wherein the file block uploading manner further comprises,
splitting the file block into a plurality of fragmented files, wherein the size of the fragmented files is smaller than or equal to the size of the file uploaded at a time;
and uploading the segmented files to the storage server in sequence according to the splitting sequence of the segmented files.
3. The file uploading method according to claim 2, wherein a plurality of adjacent fragmented files have partially identical contents, and the content identical rate between the adjacent fragmented files is at least 50%.
4. The method of uploading files according to claim 1, wherein determining the predetermined threshold value further comprises, based on each initial threshold value and the uploading time of the first test file and the second test file corresponding to each initial threshold value,
respectively drawing uploading curves of the first test file and the second test file by taking the initial threshold value as an abscissa and the uploading time as an ordinate;
determining an intersection point of uploading curves of the first test file and the second test file;
and taking the initial value of the threshold corresponding to the intersection point as the preset threshold.
5. A document uploading apparatus, comprising:
the file to be uploaded receiving unit is used for receiving the file to be uploaded sent by the client;
an uploading mode determining unit, configured to determine an uploading mode of the file to be uploaded according to the size of the file to be uploaded and a predetermined threshold, where the predetermined threshold determines test results of multiple uploading tests on multiple test files with different sizes according to multiple uploading modes;
the file to be uploaded uploading unit is used for uploading the file to be uploaded to a storage server according to the uploading mode;
the uploading mode comprises a file stream uploading mode and a file block uploading mode;
the file stream uploading mode comprises the steps of caching the file to be uploaded in a file receiving cache area in a memory, and uploading the file to be uploaded cached in the file receiving cache area to the storage server;
the file block uploading mode comprises the steps of generating a file block according to the file to be uploaded, caching the file block in a file caching unit in a disk, and uploading the file block cached in the file caching unit to the storage server;
the step of determining the predetermined threshold value comprises,
Step 1: determining an initial threshold according to the size of the system memory;
step 2: determining the size of a first test file and the size of a second test file according to the initial threshold value so as to upload the first test file to the storage server in the file stream uploading mode, upload the second test file to the storage server in the file block uploading mode, and respectively record uploading time of the first test file and the second test file;
step 3: adjusting the initial threshold according to a preset step length, and repeating the step 2 to respectively obtain uploading time of the first test file and the second test file corresponding to each initial threshold;
step 4: determining the preset threshold according to each initial threshold and the uploading time of the first test file and the second test file corresponding to each initial threshold;
determining the size of the first test file and the size of the second test file based on the initial threshold includes,
calculating the difference between the initial threshold value and a preset adjustment amount to obtain the size of the first test file, wherein the preset adjustment amount is a positive number;
and calculating the sum of the initial threshold value and the preset adjustment quantity to obtain the size of the second test file.
6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 4 when executing the computer program.
7. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method of any of claims 1 to 4.
8. A computer program product, characterized in that it comprises a computer program which, when executed by a processor, implements the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210454903.1A CN114827130B (en) | 2022-04-24 | 2022-04-24 | File uploading method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210454903.1A CN114827130B (en) | 2022-04-24 | 2022-04-24 | File uploading method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114827130A CN114827130A (en) | 2022-07-29 |
| CN114827130B true CN114827130B (en) | 2024-04-16 |
Family
ID=82509301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210454903.1A Active CN114827130B (en) | 2022-04-24 | 2022-04-24 | File uploading method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114827130B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109995821A (en) * | 2017-12-29 | 2019-07-09 | 中移(苏州)软件技术有限公司 | Method and system, the client, server, object storage system of file upload |
| CN110247986A (en) * | 2019-06-28 | 2019-09-17 | 北京奇艺世纪科技有限公司 | A kind of document transmission method, device and electronic equipment |
| CN113055455A (en) * | 2021-03-09 | 2021-06-29 | 中国联合网络通信集团有限公司 | File uploading method and equipment |
| CN113315841A (en) * | 2021-05-31 | 2021-08-27 | 平安科技（深圳）有限公司 | File uploading method and device, medium and electronic equipment |
-
2022
- 2022-04-24 CN CN202210454903.1A patent/CN114827130B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109995821A (en) * | 2017-12-29 | 2019-07-09 | 中移(苏州)软件技术有限公司 | Method and system, the client, server, object storage system of file upload |
| CN110247986A (en) * | 2019-06-28 | 2019-09-17 | 北京奇艺世纪科技有限公司 | A kind of document transmission method, device and electronic equipment |
| CN113055455A (en) * | 2021-03-09 | 2021-06-29 | 中国联合网络通信集团有限公司 | File uploading method and equipment |
| CN113315841A (en) * | 2021-05-31 | 2021-08-27 | 平安科技（深圳）有限公司 | File uploading method and device, medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114827130A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7924884B2 (en) | Performance logging using relative differentials and skip recording | |
| US10374928B1 (en) | Efficient bandwidth estimation | |
| US20120011101A1 (en) | Integrating client and server deduplication systems | |
| JP2020509490A (en) | Sequential storage of data in zones in a distributed storage network | |
| US7539762B2 (en) | Method, system and program product for determining an initial number of connections for a multi-source file download | |
| US9578074B2 (en) | Adaptive content transmission | |
| CN111147564A (en) | Data file transmission method, system and communication terminal | |
| US9954979B2 (en) | Protocol selection for transmission control protocol/internet protocol (TCP/IP) | |
| CN112492033A (en) | File transmission method, system and computer readable storage medium | |
| US10484730B1 (en) | Chunked transfer mode bandwidth estimation | |
| CN110493342B (en) | File transmission method and device, electronic equipment and readable storage medium | |
| US20210373776A1 (en) | System and method for data compaction and security using multiple encoding algorithms | |
| CN103152606B (en) | Video file processing method and device, system | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| CN114827130B (en) | File uploading method and device | |
| US20090248845A1 (en) | Network bandwidth control for network storage | |
| CN111835801B (en) | File downloading method, device, server, edge equipment, terminal and medium | |
| GB2567149A (en) | Managing data Compression | |
| US10769016B2 (en) | Storing a plurality of correlated data in a dispersed storage network | |
| Julián-Moreno et al. | On the feasibility of 40 gbps network data capture and retention with general purpose hardware | |
| US10673801B2 (en) | Dynamic communication session management | |
| CN115878512A (en) | Memory mapping method, system, equipment and storage medium | |
| CN102891732A (en) | Data sending method and device and data receiving method and device | |
| US20180246666A1 (en) | Methods for performing data deduplication on data blocks at granularity level and devices thereof | |
| US20220067212A1 (en) | Offloading operations from a primary processing device to a secondary processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |