CN116366629A - File uploading and acquiring method, computer equipment and storage medium - Google Patents
File uploading and acquiring method, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116366629A CN116366629A CN202211666029.4A CN202211666029A CN116366629A CN 116366629 A CN116366629 A CN 116366629A CN 202211666029 A CN202211666029 A CN 202211666029A CN 116366629 A CN116366629 A CN 116366629A
- Authority
- CN
- China
- Prior art keywords
- file
- identification information
- sub
- block
- blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 95
- 230000015654 memory Effects 0.000 claims description 36
- 230000004044 response Effects 0.000 claims description 24
- 238000012795 verification Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 6
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 24
- 238000004364 calculation method Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 239000007787 solid Substances 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 238000003491 array Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention discloses a file uploading and acquiring method, a computer device and a storage medium, which are executed by a client, wherein the method comprises the following steps: when the file information of the file to be uploaded is larger than a preset threshold value, dividing the file to be uploaded into a plurality of file blocks according to the file information and the preset threshold value, and generating sub-identification information corresponding to each file block; and sending each file block and sub-identification information corresponding to the file block to a server so that the server can sequentially combine each file block according to the sub-identification information to obtain the file to be uploaded.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method for uploading and obtaining a file, a computer device, and a storage medium.
Background
With the development of internet technology and the popularization of big data application, the web application for each industry is various, and the file uploading function basically belongs to the necessary function. But when very large files, such as operating system images, large software installation packages, and large volumes of files are encountered for uploading, they can be very slow, or cause a large number of web-side links to tie up and hang up server processing links, or because of the long transmission volume being forced to shut down by nodes in the network. Therefore, a new file uploading method needs to be proposed to solve the problems existing in the large file uploading process in the prior art.
Disclosure of Invention
Therefore, in order to solve the defects in the prior art, the embodiment of the invention provides a file uploading and acquiring method, a computer device and a storage medium.
According to a first aspect, an embodiment of the present invention discloses a file uploading method, which is executed by a client, and includes:
when the file information of the file to be uploaded is larger than a preset threshold value, dividing the file to be uploaded into a plurality of file blocks according to the file information and the preset threshold value, and generating sub-identification information corresponding to each file block;
and sending each file block and the sub-identification information corresponding to the file block to a server so that the subsequent server can sequentially combine each file block according to the sub-identification information to obtain the file to be uploaded.
Optionally, the sub-identification information includes first sub-identification information and second sub-identification information, and the second sub-identification information of the previous file block in the adjacent file block is the same as the first sub-identification information of the next file block.
Optionally, each file block and sub-identification information corresponding to the file block are sent to a server, which specifically includes:
all the file blocks and sub-identification information corresponding to the file blocks are simultaneously sent to a server;
or (b)
Transmitting sub-identification information corresponding to a first file block to a server, wherein the first file block is any one of a plurality of file blocks;
obtaining a response instruction fed back by the server according to the sub-identification information corresponding to the first file block, wherein the response instruction is generated after the server determines whether the first file block is received according to the sub-identification information of the first file block;
when the response instruction indicates that the server does not receive the first file block, the file block is sent to the server.
Optionally, the method further comprises:
and sending the file blocks and the subfile identifications corresponding to each file block to a server in a multithreaded polling mode.
Optionally, the first total identification information is generated according to sub identification information of all file blocks; after sending each file block and the sub-identification information corresponding to the file block to the server, the method further comprises:
acquiring second total identification information fed back by the server, wherein the second total identification information is generated according to all file blocks received by the server and sub-identification information corresponding to each file block in all received file blocks;
comparing the first total identification information with the second total identification information;
If the second total identification information is different from the first total identification information, uploading all the file blocks and the sub identification information corresponding to each file block again.
According to a second aspect, an embodiment of the present invention discloses a file acquisition method, which is executed by a server, and includes:
acquiring at least one file block of a file to be uploaded and sub-identification information corresponding to each file block, which are sent by a client at the current moment;
and sequentially combining the acquired file blocks according to the sub-identification information corresponding to the file blocks and the file blocks.
Optionally, obtaining at least one file block of a file to be uploaded and sub-identification information corresponding to each file block, which are sent by a client at the current moment, specifically includes:
simultaneously acquiring the file block sent by the client and sub-identification information corresponding to the file block;
or (b)
Sub-identification information corresponding to a first file block sent by a client is obtained;
inquiring whether a file block corresponding to the sub-identification information exists in the server, and if the file block corresponding to the sub-identification information does not exist in the server, sending a response instruction to the client;
and acquiring a first file block sent by the client according to the response instruction.
Optionally, sequentially combining the obtained file blocks according to the sub-identification information corresponding to the file blocks and the file blocks, including:
Sequencing all file blocks according to the file block sequence to obtain an uploading file;
determining second total identification information of the uploaded file according to all the file blocks and sub-identification information corresponding to the file blocks;
transmitting the second total identification information to the client;
and acquiring all file blocks and sub-identification information corresponding to each file block sent by the client after the verification of the second total identification information fails until the second total identification information of the uploaded file is successfully verified.
According to a third aspect, an embodiment of the present invention discloses a file uploading device, applied to a client, the device includes:
the file dividing module is used for dividing the file to be uploaded into a plurality of file blocks according to the file information and the preset threshold value when the file information of the file to be uploaded is larger than the preset threshold value, and generating sub-identification information corresponding to each file block;
the file sending module is used for sending each file block and the sub-identification information corresponding to the file blocks to the server, so that the subsequent server can sequentially combine each file block according to the sub-identification information to obtain the file to be uploaded.
According to a fourth aspect, an embodiment of the present invention discloses a file acquisition device, applied to a server, the device including:
The acquisition module is used for acquiring at least one file block of the file to be uploaded and the sub-identification information corresponding to each file block, which are sent by the client at the current moment;
and the combination module is used for sequentially combining the acquired file blocks according to the sub-identification information corresponding to the file blocks and the file blocks.
According to a fifth aspect, an embodiment of the present invention further discloses a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform steps of the file upload method as in the first aspect or any alternative embodiment of the first aspect.
According to a sixth aspect, an embodiment of the present invention also discloses a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the file upload method as in the first aspect or any alternative embodiment of the first aspect.
According to a seventh aspect, an embodiment of the present invention further discloses a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform steps of a file retrieval method as in the second aspect or any alternative embodiment of the second aspect.
According to an eighth aspect, an embodiment of the present invention also discloses a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the file acquisition method as in the second aspect or any of the alternative embodiments of the second aspect.
The technical scheme of the invention has the following advantages:
according to the file uploading and acquiring method, in the file uploading process, when the file information of the file to be uploaded is larger than a preset threshold value, the fact that the file to be uploaded is too large at the moment is indicated, the file to be uploaded is directly uploaded and possibly has downtime is indicated, and according to the preset threshold value and the file information, a client divides the file to be uploaded into a plurality of file blocks, and sub-identification information corresponding to each file block is generated for identifying the information of each file block; further, all the file blocks and the sub-identification information corresponding to the file blocks are sent to the server, so that the file blocks to be uploaded are uploaded to the server, and the risks of downtime in the large file uploading process and the risks of forced shutdown by the network transfer node under the condition of large transmission quantity in a long time are avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a file uploading and acquiring system according to an embodiment of the present invention
FIG. 3 is a schematic diagram of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 4 is a flowchart showing a specific example of a file acquisition method according to an embodiment of the present invention;
FIG. 5 is a flowchart of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 6 is a flowchart of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 7 is a flowchart of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 8 is a flowchart of a specific example of a file uploading method according to an embodiment of the present invention;
FIG. 9 is a schematic block diagram of a specific example of a file uploading device in an embodiment of the present invention;
FIG. 10 is a schematic block diagram of a specific example of a file acquisition device in accordance with an embodiment of the present invention;
FIG. 11 is a diagram illustrating a specific example of a computer device in a client according to an embodiment of the present invention;
fig. 12 is a diagram showing a specific example of a computer device in a server according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
Aiming at the technical problems mentioned in the background art, the embodiment of the application provides a file uploading method, and particularly referring to fig. 1, a flowchart of the file uploading method is provided in the embodiment of the invention. Before introducing the implementation steps of the method, a system for uploading and acquiring files, which is applicable to the method, is first described, as shown in fig. 2, and is an application scene diagram of a specific embodiment of the present invention. In the file uploading and acquiring system in the embodiment of the invention, the client can be a corresponding webpage in a browser and the like in the file uploading process, the file to be uploaded can be directly acquired, and the server can be one end for receiving the file in the file acquiring process.
The specific execution process of each module in the file uploading and acquiring system is described next.
Specifically, referring to fig. 1, the file uploading method is executed by a client, and includes the following steps:
step 101, when the file information of the file to be uploaded is greater than a preset threshold, dividing the file to be uploaded into a plurality of file blocks according to the file information and the preset threshold, and generating sub-identification information corresponding to each file block.
In the uploading process, when the file information of the file to be uploaded is greater than a preset threshold, the file to be uploaded needs to be divided into a plurality of file blocks, and sub-identification information corresponding to each file block needs to be generated for combining each individual file block to obtain a complete file to be uploaded in order to determine the content information of the file blocks and the relation between the file blocks. Specifically, the sub-identification information may be an order and a number of corresponding file blocks, belonging to a unique identification of the file blocks.
The file to be uploaded is divided into a plurality of file blocks, the safety of the file to be uploaded can be improved in the uploading process, and when one file block or a plurality of file blocks are lost, the complete file to be uploaded cannot be obtained as long as all the file blocks are not lost. Even if all the file blocks are acquired, the complete file to be uploaded cannot be obtained by combining under the condition that the corresponding sub-file identifiers are not known, so that the safety in the file uploading process can be further improved.
In general, the definition of a large file may be that an individual file exceeds Gb, or when package downloading is performed, all files or some files with sizes exceeding Gb may be referred to as large files, and of course, in the embodiment of the present invention, the specific numerical value of the preset threshold is not limited, and may be determined by a person skilled in the art according to practical situations.
In addition, another explanation is provided for the preset threshold, in the preset threshold, the client can smoothly upload the file to be uploaded, when the preset threshold is exceeded, the file is uploaded, and the situation that the background art is crashed or the node is turned off can possibly occur, and at the moment, the file to be uploaded can be divided.
In a preferred embodiment, the sub-identification information includes first sub-identification information and second sub-identification information, and the second sub-identification information of the previous file block in the adjacent file block is identical to the first sub-identification information of the subsequent file block.
The hash value obtained by calculation can be used as the identification information corresponding to the file block, and the hash value of the file block corresponds to a string of data, so that the initial value and the end value of the hash value are included, and the initial value and the end value are also used as the sub-identification information of the file block, thereby further improving the security of the file block.
Step 102, transmitting each file block and sub-identification information corresponding to the file block to a server, so that the subsequent server sequentially combines each file block according to the sub-identification information to obtain a file to be uploaded.
For example, after the file to be uploaded is divided into a plurality of file blocks and sub-identification information corresponding to each file block is generated, each file block and the corresponding sub-identification information are sent to the server to complete uploading of the file to be uploaded, and an uploading method in the prior art may be adopted in the uploading process, which is not repeated here. The subsequent server side can combine the file blocks according to the sub-identification information corresponding to the file blocks, so that the corresponding file to be uploaded is obtained.
In a preferred embodiment, considering how to send a huge number of file blocks to a server, if the file blocks are sent sequentially, and the time for sending the file to be uploaded to the server is not reduced, in order to increase the uploading speed of the file blocks, multiple file blocks and the subfile identification corresponding to each file block can be sent to the server in a multithreaded polling mode, or distributed uploading is adopted, and when the multithreaded uploading or the distributed uploading is carried out, the uploading of one file block is finished, the next file block is continuously selected, and the uploading is continued until all the file blocks are uploaded.
Specifically, when the application performs file uploading in the browser, the requirements of the communication protocol interface standard are higher, as shown in fig. 3, the timeout setting and the size limit can be set, and in the distributed scenario, all requests have a mandatory timeout response time, which is typically 5 to 50 seconds.
In a preferred embodiment, considering that the multi-thread sending of the file blocks is considered, in the sending process, the file blocks possibly exist are sent to the server, the file blocks possibly exist are not sent to the server, so that the file blocks possibly exist are sent multiple times, and the file blocks are not sent to the server all the time, so that each file block and sub-identification information corresponding to the file block are sent to the server, and specifically two modes can be included, wherein the first mode is that: and simultaneously transmitting all the file blocks and the sub-identification information corresponding to the file blocks to a server.
The second way is: transmitting sub-identification information corresponding to a first file block to a server, wherein the first file block is any one of a plurality of file blocks; obtaining a response instruction fed back by the server according to the sub-identification information corresponding to the first file block, wherein the response instruction is generated after the server determines whether the first file block is received according to the sub-identification information of the first file block; when the response instruction indicates that the server does not receive the first file block, the file block is sent to the server.
In the first mode, for example, all the file blocks and the sub-identification information corresponding to the file blocks are sent to the server at the same time, that is, all the file blocks and the sub-identification information are sent only once, so that all the file blocks can be ensured to be sent to the server, and the above problems are avoided.
In the second mode, before sending the file block, sub-identification information corresponding to the file block is sent to the server, so that when no corresponding sub-identification information exists in the server, it is stated that the file block is not sent to the server before, and after corresponding response information is obtained, the corresponding file block is sent to the server, so that the above problem is avoided.
Similarly, when the packaged files are transmitted in a large scale, before the corresponding files to be uploaded are transmitted, the identification information of the files to be uploaded can be transmitted to the server to inquire whether the files are transmitted, so that repeated uploading of the files can be ensured, network resources are occupied, and the file uploading speed can be further improved.
On the basis of the above embodiment, the client needs to verify whether the server has received the complete file to be uploaded, and whether the combination sequence of the files of the server is correct. The method further includes the steps that after the client generates identification information of the file to be uploaded according to the file to be uploaded, the first total identification information is used as the identification information of the file to be uploaded, the first total identification information is generated according to sub-identification information of all file blocks, and therefore each file block and the sub-identification information corresponding to the file block are sent to the server:
Acquiring second total identification information fed back by the server, wherein the second total identification information is generated according to all file blocks received by the server and sub-identification information corresponding to each file block in all received file blocks; comparing the first total identification information with the second total identification information; if the second total identification information is different from the first total identification information, uploading all the file blocks and the sub identification information corresponding to each file block again.
The second total identification information is identification information generated by the server according to sub-identification information corresponding to the obtained file blocks, and the second total identification information can reflect whether the order of the combination of the file blocks on the server side is correct, and the second total identification information corresponds to the first total identification information under the condition that the combination is correct, if the combination is not correct, the condition that the combination of the file blocks is incorrect, or the condition that transmission is interrupted and missed occurs when the file blocks are uploaded is indicated, so that the server can obtain a complete file to be uploaded, and all the file blocks and the sub-identification information corresponding to each file block are uploaded again until the obtained second total identification information corresponds to the first total identification information.
According to the file uploading method, in the file uploading process, when the file information of the file to be uploaded is larger than the preset threshold value, the situation that the file to be uploaded is too large at the moment and downtime is likely to occur in the process of directly uploading the file is indicated, and aiming at the situation, a client divides the file to be uploaded into a plurality of file blocks according to the preset threshold value and the file information, and sub-identification information corresponding to each file block is generated for identifying the information of each file block; further, all the file blocks and the sub-identification information corresponding to the file blocks are sent to the server, so that the file blocks to be uploaded are uploaded to the server, and the risks of downtime in the large file uploading process and the risks of forced shutdown by the network transfer node under the condition of large transmission quantity in a long time are avoided.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a file obtaining method, as shown in fig. 4, executed by a server, where the method includes the following steps:
step 401, obtaining at least one file block of a file to be uploaded and sub-identification information corresponding to each file block, wherein the at least one file block and the sub-identification information correspond to each file block are sent by a client at the current moment.
The method includes the steps of obtaining a file block sent by a client and sub-identification information corresponding to the file block, and specifically, obtaining the file block sent by the client and the corresponding sub-identification information after the client establishes communication connection with a server.
Specifically, in an alternative embodiment, the manner in which the file blocks are sent for the client may be obtained in two ways, the first way: and simultaneously acquiring the file block sent by the client and sub-identification information corresponding to the file block. The second way is: sub-identification information corresponding to a first file block sent by a client is obtained; inquiring whether a file block corresponding to the sub-identification information exists in the server, and if the file block corresponding to the sub-identification information does not exist in the server, sending a response instruction to the client; and acquiring a first file block sent by the client according to the response instruction.
For the first way, the client sends the file block to the server together with the corresponding sub-identification information. The second mode is that the client side firstly sends the sub-identification information corresponding to the file blocks to the server, after the server obtains the sub-identification information, the server inquires whether the file blocks corresponding to the sub-identification information exist locally, when the file blocks exist, the corresponding file blocks are already obtained before the explanation, the re-obtaining is equivalent to the repeated obtaining, and then the response instruction needs to be sent to the client side to explain that the file blocks exist and do not need to be sent again, so that network resources occupied by repeated sending are reduced. When it is determined that the corresponding file block does not exist locally at the server, the corresponding file block may continue to be transmitted to the server.
And step 402, sequentially combining the acquired file blocks according to the sub-identification information corresponding to the file blocks and the file blocks.
For example, after the file blocks and the corresponding sub-identification information are obtained, the corresponding file blocks may be combined in real time according to the sub-identification information, each time a file block is obtained, the file block is combined with a previous file block according to the corresponding sub-identification information, for example, the sub-identification information of the file block obtained at the current moment is 4, the sub-identification information of the uploaded file formed by the obtained file blocks is respectively 1, 2 and 6, and the sub-identification information of the uploaded file formed by the obtained file blocks is sequentially ordered and combined, so that the file block with the sub-identification information of 4 and the sub-identification information of the previous file block need to be ordered at this time, and the corresponding file blocks can be 1, 2, 4 and 6, that is, each time a file block is obtained, is sequentially combined with the previous file block, so as to obtain a new uploaded file, until all the file blocks are obtained.
When judging whether to acquire all the file blocks, it may be judged that all the file blocks are acquired when the file blocks sent by the client are not acquired within a preset time, or the client may send an instruction that the file blocks are already transmitted to the server, etc. in this embodiment, the mode of determining that the file blocks are acquired is not limited, and those skilled in the art may determine according to actual situations.
Therefore, the positions of the file blocks are calculated in real time, and the problems that a large amount of calculation amount is needed and the system is halted or time is consumed can be avoided when all the file blocks are obtained subsequently, calculated and combined sequentially.
In an alternative embodiment, sequentially combining the obtained file blocks according to the sub-identification information corresponding to the file blocks and the file blocks includes:
sequencing all file blocks according to the file block sequence to obtain an uploading file;
determining second total identification information of the uploaded file according to all the file blocks and sub-identification information corresponding to the file blocks;
transmitting the second total identification information to the client;
and acquiring all file blocks and sub-identification information corresponding to each file block sent by the client after the verification of the second total identification information fails until the second total identification information of the uploaded file is successfully verified.
For example, after the file blocks and the sub-identification information corresponding to the file blocks are obtained, the file blocks are sequentially combined to obtain an uploaded file, and in order to verify whether the uploaded file corresponds to the client one by one, the uploaded file needs to be verified to the client.
The specific verification method may be that after the uploaded file is obtained, second total identification information corresponding to the uploaded file is generated according to the uploaded file, the second total identification information corresponds to the uploaded file one by one, the second total identification information is sent to a client for verification, first total identification information corresponding to the file to be uploaded one by one is provided in the client, when the client verifies that the second total identification information is correct according to the first total identification information, a verification success instruction is sent to a server, and after the server obtains the verification success instruction, the uploaded file can be determined to be the file to be uploaded.
When the server does not acquire the verification success instruction, the client is not verified successfully, at the moment, the client can send all the file blocks to the server again, the server directly acquires all the file blocks sent by the client, and after the file blocks are acquired, the acquired file blocks are sequentially combined again until the finally acquired second total identification information of the uploaded file is verified successfully at the client.
Specifically, the sub-identification information of the file block may be a hash value corresponding to each file block, and the second total identification information corresponding to the sub-identification information may be a hash value of the entire uploaded file, where the first total identification information is a hash value of the file to be uploaded calculated by the client, that is, the sub-identification information (hash values) of all the file blocks are added up to be equal to the first total identification information, so that when the client verifies the second total identification information, the second total identification information is equal to the first total identification information in an ideal case. By adopting the scheme of hash value calculation, the calculation time of each file block is consistent, and the calculation performance is doubled. The common hash algorithm (md 5 and sha 1) calculates the 1G file for about 4 seconds, and if the actual environment multithreading contends for the scene of the local file system IO is considered, the 1G file takes about 8-10 seconds to complete the calculation. The scheme of the method and the device can save the corresponding time of calculation because of real-time incremental calculation. And meanwhile, the risk of response timeout of the last file block can be fundamentally eliminated.
When the corresponding file blocks are sequentially combined according to the sub-identification information, the relation between the start value and the end value of the hash value of each file block can be calculated in real time, and the sequence relation can be determined according to the start value of the file block and the end value of the acquired file block after each file block is acquired.
In this way, in the process of acquiring the file, at least one file block and sub-identification information corresponding to the file block, which are sent by the client at the current moment, are acquired, the sequence relation between the file blocks acquired at the current moment is determined according to the sub-identification information and the corresponding file blocks, and the uploaded file is determined, so that the problem of the sudden increase of calculated amount caused by the centralized calculation of the sequence among the file blocks after all the file blocks are finally acquired is avoided, and meanwhile, the risk of downtime in the uploading process of the large file and the risk of forced shutdown by a network transfer node under the condition of large transmission amount with long time are avoided.
As shown in fig. 5, which is a flowchart of a specific embodiment of the file uploading and obtaining method, when uploading files in batches, a browser (client) scans files to be uploaded and paths of the files to be uploaded, and blocks the files larger than a preset threshold value to obtain a plurality of file blocks; before uploading the file and the file block, sending corresponding sub-identification information (check code) to the server to judge whether the file is uploaded to the server, uploading the file by adopting a multithreading mode when the file is determined to be not uploaded, judging whether the file block which is not uploaded exists after each thread is processed, and continuing uploading the next file if the file block which is not uploaded exists; after all the file blocks are uploaded, verifying whether the second total identification information (check value) of the server is consistent with the first total identification information of the client, if not, uploading again, and if so, starting uploading of the next file until all the files are uploaded.
In the following, an uploading process is described as an example in a specific embodiment.
Providing an option entry in the web page, and directly loading a folder from the page, wherein the folder often contains tens of thousands of files; as shown in fig. 6 and 7, selecting a target directory node to be uploaded, and selecting a local folder to be uploaded; the web page scans files on a local path through an incoming folder path, acquires an uploaded file name list, checks information such as file size after scanning summarized files, performs division processing on the files, performs hash calculation on the whole files, and sends out a request to judge whether the files exist on a server or not, and the specific implementation method is as follows:
after the file splitting and uploading overall task is decomposed into hundreds of thousands of subtasks, the front-end webpage distribution thread executes the tasks, namely starting block-by-block file uploading; starting from the current node, entering a single-thread uploading file logic flow, uploading file blocks in single-thread sequential processing, and calculating sub-identification information uploading files block by block, specifically:
and uploading and returning by a single thread, wherein the first sub-identification information and the second sub-identification information are consistent.
In the process of uploading and acquiring mass files, a server side designs a data checking mechanism in order to prevent the disorder of uploading file blocks. In the conventional data verification, the whole file is verified integrally or the block-by-block file is verified independently. According to the scheme, the whole file block and the single file block are considered, the hash_init is used as first sub-identification information for file block verification, after initialization, the whole file block is verified to generate the hash_final which is used as second sub-identification information for file block ending, and meanwhile, the hash_final is also used as first sub-identification information for next file block verification.
Finally, when the file block is verified, the server calculates the verification value of the final file all the time, so that the stability of data calculation in the whole uploading period is ensured, the calculation of the verification value after the file is uploaded by the traditional scheme easily causes the problem that the operand is suddenly increased and the uploading flow of the file is blocked.
The scheme effectively solves the performance contradiction caused by file data loss, file block verification and file integral verification in the uploading process.
The single thread judges whether the current file is uploaded, if the current file is not uploaded to the next file block, if the file is uploaded, the next file is selected to be uploaded continuously in the thread. If the uploading of all the files is finished, the front-end interface uploading thread pool is released.
As shown in fig. 8, in order to use hash codes to mark file blocks, a specific implementation process may be that a file to be uploaded is a hash code total value, corresponding to first total identification information, the hash code total value is divided into N file blocks with a preset threshold value as an interval on the basis of the hash code total value, each file block 512bit corresponds to a hash code having a start value and an end value, the end value of the last file block corresponds to the start value of the next file block, so that all the file blocks can be combined to obtain a hash code total value (final hash value) according to the combined file blocks, specifically may include a total start value and a total bundle value, and when the hash code total value (first total identification information) of the combined file blocks is consistent with the second total identification information, the file uploading at this time is described.
In the foregoing, several embodiments of the method for uploading and obtaining a file provided in the present application are described below to illustrate other embodiments for uploading and obtaining a file provided in the present application, which are specifically referred to as follows.
The embodiment of the invention also discloses a file uploading device, which is applied to the client as shown in fig. 9, and comprises:
the file dividing module 901 is configured to divide a file to be uploaded into a plurality of file blocks according to the file information and a preset threshold when the file information of the file to be uploaded is greater than the preset threshold, and generate sub-identification information corresponding to each file block;
the file sending module 902 is configured to send each file block and sub-identifier information corresponding to the file block to the server, so that the subsequent server sequentially combines each file block according to the sub-identifier information to obtain a file to be uploaded.
As an optional real-time manner of the present invention, the sub-identification information includes first sub-identification information and second sub-identification information, and the second sub-identification information of the previous file block in the adjacent file block is identical to the first sub-identification information of the next file block.
As an optional real-time manner of the present invention, sub-identification information corresponding to each file block is sent to a server, which is specifically configured to:
All the file blocks and sub-identification information corresponding to the file blocks are simultaneously sent to a server; or (b)
Transmitting sub-identification information corresponding to a first file block to a server, wherein the first file block is any one of a plurality of file blocks;
obtaining a response instruction fed back by the server according to the sub-identification information corresponding to the first file block, wherein the response instruction is generated after the server determines whether the first file block is received according to the sub-identification information of the first file block;
when the response instruction indicates that the server does not receive the first file block, the file block is sent to the server.
As an alternative real-time manner of the invention, the device is also used for:
and sending the file blocks and the subfile identifications corresponding to each file block to a server in a multithreaded polling mode.
As an optional real-time mode of the invention, the first total identification information is generated according to the sub identification information of all the file blocks; after sending each file block and the sub-identification information corresponding to the file block to the server, the device is further configured to:
acquiring second total identification information fed back by the server, wherein the second total identification information is generated according to all file blocks received by the server and sub-identification information corresponding to each file block in all received file blocks;
Comparing the first total identification information with the second total identification information;
if the second total identification information is different from the first total identification information, uploading all the file blocks and the sub identification information corresponding to each file block again.
The functions executed by each component in the file uploading device provided in the embodiment of the present invention are described in detail in any of the above method embodiments, so that a detailed description is omitted here.
By executing the device, in the process of uploading the file, when the file information of the file to be uploaded is larger than a preset threshold value, the situation that downtime is likely to occur in uploading the file is indicated to be too large, and aiming at the situation, the client divides the file to be uploaded into a plurality of file blocks according to the preset threshold value and the file information, and sub-identification information corresponding to each file block is generated for identifying the information of each file block; further, all the file blocks and the sub-identification information corresponding to the file blocks are sent to the server, so that the file blocks to be uploaded are uploaded to the server, and the risks of downtime in the large file uploading process and the risks of forced shutdown by the network transfer node under the condition of large transmission quantity in a long time are avoided.
The embodiment of the invention discloses a file downloading method, which is applied to a server, as shown in fig. 10, and comprises the following steps:
an obtaining module 1001, configured to obtain at least one file block of a file to be uploaded and sub-identification information corresponding to each file block, where the sub-identification information is sent by a client at a current moment;
and a combination module 1002, configured to sequentially combine the obtained file blocks according to the sub-identification information corresponding to the file blocks and the file blocks.
As an optional real-time manner of the present invention, at least one file block of a file to be uploaded, which is sent by a client at the current moment, and sub-identification information corresponding to each file block are obtained, which is specifically used for:
simultaneously acquiring the file block sent by the client and sub-identification information corresponding to the file block; or (b)
Sub-identification information corresponding to a first file block sent by a client is obtained;
inquiring whether a file block corresponding to the sub-identification information exists in the server, and if the file block corresponding to the sub-identification information does not exist in the server, sending a response instruction to the client;
and acquiring a first file block sent by the client according to the response instruction.
As an optional real-time manner of the present invention, according to the sub-identification information corresponding to the file blocks and the file blocks, the obtained file blocks are sequentially combined, which is specifically used for:
Sequencing all file blocks according to the file block sequence to obtain an uploading file;
determining second total identification information of the uploaded file according to all the file blocks and sub-identification information corresponding to the file blocks;
transmitting the second total identification information to the client;
and acquiring all file blocks and sub-identification information corresponding to each file block sent by the client after the verification of the second total identification information fails until the second total identification information of the uploaded file is successfully verified.
The functions executed by each component in the file obtaining apparatus provided in the embodiment of the present invention are described in detail in any of the above method embodiments, so that a detailed description is omitted here.
By executing the device, at least one file block and sub-identification information corresponding to the file block, which are sent by a client at the current moment, are acquired in the process of acquiring the file, the sequence relation between the file blocks acquired at the current moment is determined according to the sub-identification information and the corresponding file blocks, and the file is uploaded, so that the problem of calculation amount burst increase caused when the sequence among the file blocks is intensively calculated after all the file blocks are finally acquired is avoided, and meanwhile, the risk of downtime in the uploading process of a large file and the risk of forced shutdown by a network transit node under the condition of large transmission amount with long time are avoided.
The embodiment of the present invention further provides a computer device, as shown in fig. 11, where the computing end may include a processor 1101 and a memory 1102, where the processor 1101 and the memory 1102 may be connected by a bus or other manner, and in fig. 11, the connection is exemplified by a bus.
The processor 1101 may be a central processing unit (Central Processing Unit, CPU). The processor 1101 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory 1102 is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the file downloading method in the embodiment of the present invention. The processor 1101 executes various functional applications of the processor and data processing, i.e., implements the file download method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 1102.
One or more modules are stored in the memory 1102 that, when executed by the processor 1101, perform the file download method in the embodiment shown in fig. 1.
The details of the above computer device may be understood correspondingly with respect to the corresponding relevant descriptions and effects in the embodiment shown in fig. 1, which are not repeated here.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
The embodiment of the present invention further provides a computer device, as shown in fig. 12, where the computing end may include a processor 1201 and a memory 1202, where the processor 1201 and the memory 1202 may be connected by a bus or other means, and in fig. 12, the connection is exemplified by a bus.
The processor 1201 may be a central processing unit (Central Processing Unit, CPU). The processor 1201 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory 1202, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the file downloading method in the embodiments of the present invention. The processor 1201 performs various functional applications of the processor and data processing, i.e., implements the file download method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 1202.
One or more modules are stored in memory 1202 that, when executed by processor 1201, perform the file download method of the embodiment shown in fig. 3.
The details of the above computer device may be understood correspondingly with respect to the relevant descriptions and effects corresponding to the embodiment shown in fig. 4, which are not repeated herein.
It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations are within the scope of the invention as defined by the appended claims.
Claims (12)
1. A method of file upload, performed by a client, the method comprising:
when the file information of the file to be uploaded is larger than a preset threshold value, dividing the file to be uploaded into a plurality of file blocks according to the file information and the preset threshold value, and generating sub-identification information corresponding to each file block;
and sending each file block and sub-identification information corresponding to the file block to a server so that the server can sequentially combine each file block according to the sub-identification information to obtain the file to be uploaded.
2. The method of claim 1, wherein the sub-identification information includes first sub-identification information and second sub-identification information, the second sub-identification information of a previous file block of the adjacent file blocks being identical to the first sub-identification information of a subsequent file block.
3. The method of claim 2, wherein the sending each of the file blocks and the sub-identification information corresponding to the file block to the server specifically includes:
Transmitting all the file blocks and sub-identification information corresponding to the file blocks to the server at the same time;
or (b)
Transmitting sub-identification information corresponding to a first file block to the server, wherein the first file block is any one of a plurality of file blocks;
acquiring a response instruction fed back by the server according to the sub-identification information corresponding to the first file block, wherein the response instruction is generated after the server determines whether the first file block is received according to the sub-identification information of the first file block;
and when the response instruction indicates that the server does not receive the first file block, the file block is sent to the server.
4. A method according to claim 3, characterized in that the method further comprises:
and sending the file blocks and the subfile identifications corresponding to each file block to the server in a multithreaded polling mode.
5. The method according to any of claims 2-4, wherein the first total identification information is generated from sub-identification information of all file blocks; after the file blocks and the sub-identification information corresponding to the file blocks are sent to the server, the method further comprises:
Acquiring second total identification information fed back by the server, wherein the second total identification information is generated according to all file blocks received by the server and sub-identification information corresponding to each file block in all received file blocks;
comparing the first total identification information with the second total identification information;
if the second total identification information is different from the first total identification information, uploading all the file blocks and the sub-identification information corresponding to each file block again.
6. A method of file acquisition performed by a server, the method comprising:
acquiring at least one file block of a file to be uploaded and sub-identification information corresponding to each file block, wherein the at least one file block and the sub-identification information correspond to each file block are sent by a client at the current moment;
and sequentially combining the acquired file blocks according to the sub-identification information corresponding to the file blocks and the file blocks.
7. The method of claim 6, wherein the obtaining at least one file block of the file to be uploaded and the sub-identification information corresponding to each file block, which are sent by the client at the current moment, specifically includes:
simultaneously acquiring the file block and sub-identification information corresponding to the file block, which are sent by the client;
Or (b)
Acquiring sub-identification information corresponding to a first file block sent by the client;
inquiring whether a file block corresponding to the sub-identification information exists in the server, and if the file block corresponding to the sub-identification information does not exist in the server, sending a response instruction to the client;
and acquiring a first file block sent by the client according to the response instruction.
8. The method according to claim 6 or 7, wherein sequentially combining the acquired file blocks according to the sub-identification information corresponding to the file blocks and the file blocks includes:
sequencing all the file blocks according to the file block sequence to obtain an uploading file;
determining second total identification information of the uploaded file according to all the file blocks and sub-identification information corresponding to the file blocks;
sending the second total identification information to the client;
and acquiring all file blocks and sub-identification information corresponding to each file block sent by the client after the verification of the second total identification information fails until the second total identification information of the uploaded file is successfully verified.
9. A computer device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the file upload method of any one of claims 1-5.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a node implements a file upload method according to any of claims 1-5.
11. A computer device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the file retrieval method of any of claims 6-8.
12. A computer readable storage medium having stored thereon a computer program, which when executed by a node implements a file acquisition method according to any of claims 6-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211666029.4A CN116366629A (en) | 2022-12-23 | 2022-12-23 | File uploading and acquiring method, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211666029.4A CN116366629A (en) | 2022-12-23 | 2022-12-23 | File uploading and acquiring method, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116366629A true CN116366629A (en) | 2023-06-30 |
Family
ID=86939505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211666029.4A Pending CN116366629A (en) | 2022-12-23 | 2022-12-23 | File uploading and acquiring method, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116366629A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117851351A (en) * | 2023-11-21 | 2024-04-09 | 北京青矩互联科技有限公司 | File storage method, device, equipment and medium |
-
2022
- 2022-12-23 CN CN202211666029.4A patent/CN116366629A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117851351A (en) * | 2023-11-21 | 2024-04-09 | 北京青矩互联科技有限公司 | File storage method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109496418B (en) | File downloading method and device, and equipment/terminal/server | |
| CN104219316A (en) | Method and device for processing call request in distributed system | |
| CN112165533B (en) | Data checking and transmitting method, device, equipment, system and storage medium | |
| CN111490963B (en) | Data processing method, system, equipment and storage medium based on QUIC protocol stack | |
| CN113824760B (en) | File concurrency block uploading method and device, electronic equipment and storage medium | |
| CN116366629A (en) | File uploading and acquiring method, computer equipment and storage medium | |
| US20170085340A1 (en) | Method and apparatus for data communication of vehicle | |
| CN111490947A (en) | Data packet transmitting method, data packet receiving method, system, device and medium | |
| CN107645357B (en) | Recovery method and device for incomplete transmitted file | |
| CN114553863B (en) | File transmission method and device, storage medium and electronic equipment | |
| CN114788199A (en) | Data verification method and device | |
| EP2445171B1 (en) | File transfer protocol client and implementing method thereof | |
| CN116095064A (en) | File downloading method, computer equipment and storage medium | |
| JP2000122938A (en) | Information processor | |
| CN111147597B (en) | File transmission method, terminal, electronic device and storage medium | |
| CN110768962A (en) | File self-checking method, system, sending end equipment and receiving end equipment | |
| CN115714805A (en) | Cross-platform communication connection method and system and electronic equipment | |
| CN112559349B (en) | Program running method and running device | |
| KR20220053386A (en) | Data communication method and apparatus for efficient file transmission | |
| CN110784518A (en) | Static resource acquisition method and device | |
| CN114268624B (en) | Version file transmission method and device and server | |
| CN114024712B (en) | Authentication method, authentication device, computer equipment and storage medium | |
| CN117579700B (en) | General micro-service processing method, system and equipment based on message queue | |
| CN114143300B (en) | Transaction request sending method and device | |
| CN111343217B (en) | Resource data downloading method and device, terminal equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |