CN112615899A - Large file transmission method, device and system - Google Patents
Large file transmission method, device and system Download PDFInfo
- Publication number
- CN112615899A CN112615899A CN202011337777.9A CN202011337777A CN112615899A CN 112615899 A CN112615899 A CN 112615899A CN 202011337777 A CN202011337777 A CN 202011337777A CN 112615899 A CN112615899 A CN 112615899A
- Authority
- CN
- China
- Prior art keywords
- file
- thread
- fragment
- fragments
- large file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 59
- 238000000034 method Methods 0.000 title claims abstract description 59
- 239000012634 fragment Substances 0.000 claims abstract description 238
- 238000013467 fragmentation Methods 0.000 claims description 22
- 238000006062 fragmentation reaction Methods 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 10
- 238000012546 transfer Methods 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 claims description 9
- 238000012795 verification Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000012163 sequencing technique Methods 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000005336 cracking Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
- G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a method, a device and a system for transmitting a large file, wherein the method is applied to a client and comprises the following steps: acquiring a large file to be transmitted; fragmenting the large file to obtain a plurality of file fragments, wherein each file fragment has a fragment identifier capable of representing the position of the file fragment in the large file; the method comprises the steps of calling a plurality of first threads, distributing file fragments to be transmitted for each first thread, uploading the distributed file fragments to a server through each first thread, and enabling the server to merge the plurality of file fragments into the large file based on fragment identification. Based on the invention, the transmission performance of the large file can be obviously improved, and the operation efficiency of the service is ensured.
Description
Technical Field
The invention relates to the technical field of file transmission, in particular to a method, a device and a system for transmitting a large file.
Background
File transmission is a common function of a business system, and for some complex business systems, the uploaded file data volume is large, for example, a large bidding document of a large bidding purchasing platform, which relates to a large file uploading function.
Therefore, how to improve the transmission performance of large files is an urgent problem to be solved.
Disclosure of Invention
In view of the above, to solve the above problems, the present invention provides a method, an apparatus and a system for transmitting a large file, and the technical scheme is as follows:
a large file transmission method is applied to a client side and comprises the following steps:
acquiring a large file to be transmitted;
fragmenting the large file to obtain a plurality of file fragments, wherein each file fragment has a fragment identifier capable of representing the position of the file fragment in the large file;
the method comprises the steps of calling a plurality of first threads, distributing file fragments to be transmitted for each first thread, uploading the distributed file fragments to a server through each first thread, and enabling the server to merge the plurality of file fragments into the large file based on fragment identification.
Preferably, the method further comprises:
and performing type verification on the large file, and after the type verification is passed, executing fragmentation on the large file to obtain a plurality of file fragments.
Preferably, the method further comprises:
encrypting each file fragment to obtain a corresponding file fragment ciphertext, and performing summary calculation on the large file to obtain a large file summary;
correspondingly, the uploading the allocated file fragments to the server by each first thread includes:
uploading file fragment ciphertexts corresponding to the distributed file fragments to a server through each first thread, determining a target first thread used for transmitting the large file abstract in the first threads, uploading the large file abstract to the server through the target first thread while uploading the file fragment ciphertexts, and enabling the server to obtain the file fragments through deciphering the file fragment ciphertexts and carrying out integrity check on the large file obtained through combination through the large file abstract.
Preferably, the uploading, by each first thread, the allocated file fragments to the server includes:
and uploading the distributed file fragments to the server side through each first thread by adopting a breakpoint continuous transmission technology.
A large file transfer device, the device comprising:
the file acquisition module is used for acquiring a large file to be transmitted;
the file fragmentation module is used for fragmenting the large file to obtain a plurality of file fragments, and each file fragment is provided with a fragment identifier capable of representing the position of the file fragment in the large file;
the first thread calling module is used for calling a plurality of first threads and distributing the file fragments to be transmitted for each first thread so as to upload the distributed file fragments to the server side through each first thread, and the server side can combine the file fragments into the large file based on the fragment identification.
A large file transmission method is applied to a server side and comprises the following steps:
monitoring a plurality of first threads of file fragments of a large file for transmission of a client;
creating a second thread corresponding to each first thread;
calling each second thread to receive the file fragments uploaded by the corresponding first thread through each second thread;
storing the received multiple file fragments to a specified path;
and merging the plurality of file fragments under the specified path into the large file based on the fragment identification of the file fragment.
Preferably, before the file fragment-based fragment identifier merges the plurality of file fragments under the specified path into a large file, the method further includes:
and under the condition that the client encrypts each file fragment and performs digest calculation on the large file, decrypting the file fragment ciphertext received by each second thread to obtain the corresponding file fragment, and performing integrity check on the large file obtained by combination through the large file digest received by a target second thread, wherein the target second thread is a second thread corresponding to a target first thread for transmitting the large file digest.
Preferably, the receiving, by each second thread, a file fragment uploaded by a corresponding first thread includes:
and receiving the file fragments uploaded by the corresponding first thread by adopting a breakpoint continuous transmission technology through each second thread.
A large file transfer apparatus, comprising:
the thread monitoring module is used for monitoring a plurality of first threads for transmitting file fragments of the large file at the client;
a thread creating module for creating a second thread corresponding to each first thread;
the second thread calling module is used for calling each second thread so as to receive the file fragments uploaded by the corresponding first thread through each second thread;
the file merging module is used for storing the received multiple file fragments to a specified path; and merging the plurality of file fragments under the specified path into the large file based on the fragment identification of the file fragment.
A large file transfer system, the system comprising: a server and at least one client.
The large file transmission method, the device and the system provided by the invention can fragment the large file to be transmitted and simultaneously transmit a plurality of file fragments in a multithreading asynchronous mode, thereby accelerating the transmission performance of the large file. Moreover, the file fragments have fragment identifiers representing positions of the file fragments in the large file, so that the server can determine the total file fragment number of the large file and the relative sequencing between the received file fragments based on the fragment identifiers, and the file fragments can be merged into the large file. Based on the invention, the transmission performance of the large file can be obviously improved, and the operation efficiency of the service is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a system architecture diagram of a large file transfer system according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for transmitting a large file according to an embodiment of the present invention;
fig. 3 is an example of fragmentation of a large file according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a large file transfer device according to an embodiment of the present invention;
FIG. 5 is a flowchart of another method of a large file transfer method according to an embodiment of the present invention;
fig. 6 is another schematic structural diagram of a large file transmission device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
The embodiment of the invention provides a large file transmission system, and the structural schematic diagram of the system is shown in fig. 1, and the system comprises a server 10 and at least one client 20.
In practical application, the large file transmission system can be any service system applying the invention, and based on the invention, the large file transmission of the service system greatly improves the performance, improves the safety and the stability in the file transmission process, and meets the requirement of the large file transmission of the service system.
The invention is not influenced by the traditional file transmission mode (such as FTP, HTTP or CIFS), maximally reduces the transmission delay and packet loss rate of large files and large data, fully utilizes network bandwidth resources, realizes the data file cooperation between various service systems and various operating system platforms, and solves the problem of isolated island of large file transmission.
The following describes a large file transfer method on the client 20 side and the server 10 side, respectively.
The embodiment of the present invention provides a large file transmission method, which is applied to a client 20, and a flow chart of the method is shown in fig. 2, and includes the following steps:
s101, obtaining a large file to be transmitted.
In the embodiment of the invention, the large file to be transmitted can be any file specified by a user, and the data volume of the large file exceeds the data volume threshold value transmitted by the corresponding service system.
S102, fragmenting the large file to obtain a plurality of file fragments, wherein each file fragment has a fragment identifier capable of representing the position of the file fragment in the large file.
In the embodiment of the present invention, when a large file is fragmented, fixed-block fragmentation may be adopted, fixed-length fragmentation may also be adopted, and certainly, other user-defined fragmentation modes may also be adopted, which is not limited in the embodiment of the present invention.
The fixed-block slicing and the fixed-length slicing are respectively explained as follows:
the total number of file fragments divided by the fixed-block fragments is specified regardless of the data size of the large file, for example, the total number of file fragments of the large file is defined as 10. Certainly, from the professional perspective, the segmentation method is too coarse and has many problems, and therefore, the segmentation algorithm can be optimized according to the service requirement to realize dynamic change.
And (3) fixed-length slicing: and (4) slicing is carried out according to the data volume of the large file, the data volume of each file slice is the same, and the uploading times of the client side are changed according to the data volume of the large file.
In addition, the file fragmentation aims at enabling a server side to merge file fragmentation uploaded by a client side into an original large file, so that each file fragmentation has a unique fragmentation identification in the embodiment of the invention, and the fragmentation identification has certain sequence and can represent the position of the file fragmentation in the large file.
For example, a large file is divided into 10 file fragments by using fixed-block fragments, referring to the fragment example shown in fig. 3, the file header of the large file is located in the file fragment a, the fragment identifier of the file fragment a is 1_10, which indicates that the file fragment a belongs to the first fragment of the large file after being fragmented, and the total number of the file fragments of the large file is 10, and correspondingly, the fragment identifier of the file fragment B is 2_10, and the fragment identifier of the file fragment C is 3_10 … …, so that even if the server does not receive the file fragments a, the file fragment B, and the file fragment C … … in this order, the server can determine the relative ordering among the file fragments based on the fragment identifiers, and perform a merge operation after receiving the 10 file fragments.
The large file may be a compressed file or an uncompressed file. When the fragmentation is performed, the large file is loaded into the binary data stream to generate a binary file of the large file, and then the fragmentation operation is completed according to the fixed-block or fixed-length segmentation mode, so that the file fragmentation is also a binary file.
In addition, in order to provide security of file transmission, in the embodiment of the present invention, before the large file is fragmented, the type of the large file may be checked.
The type check comprises suffix check of the file name and format check of the file header, because suffixes of the file name can be mutually converted, and after suffix check is passed, format check is further carried out on the file header which cannot be converted, so that the comprehensiveness of the check can be ensured.
For fixed services, a file type of a fixed format can be specified when a file is uploaded, and suffixes of file names and formats of file headers are respectively judged to determine whether the file types meet the specified file types.
S103, calling a plurality of first threads, distributing the file fragments to be transmitted for each first thread, uploading the distributed file fragments to a server through each first thread, and enabling the server to merge the plurality of file fragments into a large file based on fragment identification.
In the embodiment of the invention, the number of the called first threads is not related to the total number of the file fragments of the large file. And only one file fragment can be uploaded by one first thread at a time.
For example, the number of the first threads is greater than or equal to the total number of the file fragments of the large file, so that a plurality of first threads are called in parallel once, and all the file fragments can be uploaded to the server side at the same time. For another example, if the number of the first threads is smaller than the total number of the file fragments of the large file, a plurality of first threads are called in parallel once, all the file fragments cannot be uploaded to the server, and the file fragments can be allocated again for the first thread with the uploading completed until all the file fragments are uploaded completely.
The server side can determine the total file fragment number of the large file and the relative sequencing between the received file fragments based on the fragment identification of the file fragments, so as to combine the received file fragments according to the equivalent sequencing, and once the received file fragments are determined to be equal to the total file fragment number of the large file, the file fragments are combined, and the binary file fragments are combined into the original large file according to the format in the file header.
In summary, the transmission performance of the large file is limited by the network broadband, and in order to solve the problem, the transmission speed of the large file can be remarkably increased by fragmenting the large file and adopting multi-thread transmission. And multithreading transmission of file fragments is adopted, so that the performance of file deletion is accelerated.
At present, the security of the file in the transmission process is not strictly encrypted, and the file stream of the file in the transmission process can be possibly tampered. In other embodiments, to ensure the security of the transmitted data and prevent the security problems of data leakage, cracking, monitoring, etc., on the basis of the large file transmission method shown in fig. 2, the method further includes the following steps:
encrypting each file fragment to obtain a corresponding file fragment ciphertext, and performing summary calculation on the large file to obtain a large file summary;
accordingly, in step S103, "upload the allocated file fragments to the server through each first thread", the following steps may be adopted:
the distributed file fragment ciphertext corresponding to the file fragment is uploaded to the server through each first thread, a target first thread used for transmitting the large file summary in the first threads is determined, the large file summary is uploaded to the server while the file fragment ciphertext is uploaded through the target first thread, the server can obtain the file fragments through decrypting the file fragment ciphertext, and integrity check is conducted on the combined large file through the large file summary.
In the embodiment of the invention, the internal data communication security management and control is strengthened, each file fragment is processed by adopting the SM4 and MS3 encryption technologies, SSL encryption transmission is used in the uploading process of the first thread, and the data security is effectively ensured.
Specifically, the content of each file fragment may be encrypted by using a symmetric algorithm of MS4 to obtain a corresponding file fragment ciphertext; furthermore, the digest algorithm of MS3 may be used to perform a digest calculation on the content of the large file, resulting in a digest of the large file, i.e., a large file digest.
When the large file digest is transmitted, one of the multiple first threads may be designated as a target first thread, and each first thread may also be used as a target first thread, that is, at this time, each first thread transmits the large file digest while transmitting the file fragment ciphertext. The embodiments of the present invention are not limited thereto.
The server receives the file fragment ciphertext transmitted by each first thread and the large file abstract simultaneously transmitted by the target first thread. And the server decrypts the file fragment ciphertext to obtain corresponding file fragments after receiving the file fragment ciphertext, and performs integrity check on the large file through the received large file abstract after combining the plurality of file fragments into the large file based on the fragment identification. Specifically, the digest algorithm of the MS3 may be used to perform digest calculation on the content of the large file to obtain a digest of the large file to be compared, and the digest of the large file is compared with the received digest of the large file, if the two digests are the same, it indicates that the large file is not tampered in the transmission process, and if the two digests are different, it indicates that the large file is tampered in the transmission process.
It should be noted that the SM3 and SM4 belong to a national encryption and decryption algorithm, and the development thereof complies with the technical standard of the national crypto authority.
The transmission time of the large file is long, and if the transmission is interrupted due to a network or other reasons, instability is brought to a service system. In some other embodiments, in order to provide stability of large file transmission, in the embodiments of the present invention, when the first thread uploads the allocated file fragments to the server, a breakpoint resume technique may be used for uploading.
The process of transmitting the large file from the client to the server is carried out in the large file transmission process, but due to network disconnection, network halt and other emergencies caused by various reasons such as lines, equipment, computers and the like in the network, the large file is repeated in the transmission process, resources, time and other unnecessary wastes are formed, and in order to deal with the emergencies, the unnecessary wastes of the resources, the time and the like are avoided, the invention adopts a breakpoint continuous transmission technology. After the network failure is recovered, the client can continue to upload the file fragments which are not transmitted, the server can determine whether the file fragments of the client are uploaded or not through the fragment identification, and if not, the server continues to receive the file fragments.
In summary, the invention utilizes various technologies of file type verification, fragmentation, multithreading, cryptographic algorithm encryption and breakpoint resuming of the large file, and can improve the transmission performance, the safety performance and the stability performance of the large file.
Based on the large file transmission method provided in the foregoing embodiment, an embodiment of the present invention provides an apparatus for executing the large file transmission method, where a schematic structural diagram of the apparatus is shown in fig. 4, and the apparatus includes:
the file obtaining module 101 is configured to obtain a large file to be transmitted.
The file fragmentation module 102 is configured to fragment a large file to obtain a plurality of file fragments, where each file fragment has a fragment identifier capable of representing a position of the file fragment in the large file.
The first thread calling module 103 is configured to call a plurality of first threads, allocate a file fragment to be transmitted to each first thread, and upload the allocated file fragment to a server through each first thread, where the server can merge the plurality of file fragments into a large file based on a fragment identifier.
Optionally, the file fragmentation module 102 is further configured to:
and performing type verification on the large file, and after the type verification is passed, performing fragmentation on the large file to obtain a plurality of file fragments.
Optionally, the file fragmentation module 102 is further configured to:
encrypting each file fragment to obtain a corresponding file fragment ciphertext, and performing summary calculation on the large file to obtain a large file summary;
correspondingly, the first thread calling module 103 that uploads the allocated file fragments to the server through each first thread is specifically configured to:
the distributed file fragment ciphertext corresponding to the file fragment is uploaded to the server through each first thread, a target first thread used for transmitting the large file summary in the first threads is determined, the large file summary is uploaded to the server while the file fragment ciphertext is uploaded through the target first thread, the server can obtain the file fragments through decrypting the file fragment ciphertext, and integrity check is conducted on the combined large file through the large file summary.
Optionally, the first thread calling module 103 that uploads the allocated file fragments to the server through each first thread is specifically configured to:
and uploading the distributed file fragments to the server side through each first thread by adopting a breakpoint continuous transmission technology.
In the embodiment of the present invention, the detailed functions of the modules may refer to the disclosure of the method embodiment, and are not described herein again.
Based on the large file transmission method provided by the above embodiment, an embodiment of the present invention provides another large file transmission method, where the method is applied to the server 10, and a flowchart of the method is shown in fig. 5, and includes the following steps:
s201, monitoring a plurality of first threads of the client for transmitting the file fragments of the large file.
In the embodiment of the invention, the service program of the service end can monitor the data transmission request of the client, namely, the service program can monitor when each first thread transmits the file fragment.
S202, a second thread corresponding to each first thread is created.
In the embodiment of the invention, when the server monitors that a first thread transmits the file fragment, a new thread corresponding to the first thread can be created to receive the file fragment transmitted by the first thread.
S203, calling each second thread to receive the file fragments uploaded by the corresponding first thread through each second thread.
And S204, storing the received multiple file fragments to a specified path.
In the embodiment of the invention, each time a file fragment is received, the file fragment is stored under the appointed path until all the file fragments of the large file are received.
S205, merging the plurality of file fragments under the appointed path into a large file based on the fragment identification of the file fragment.
In the embodiment of the invention, based on the fragment identifier of the file fragment, the server can determine the total file fragment number of the large file, and if the number of the file fragments under the specified path is equal to the total file fragment number of the large file, the transmission of the large file is determined to be finished, and at this time, the file fragments are merged. For specific details, reference may be made to the parts disclosed in the foregoing embodiments of the method for transmitting a large file, and details are not described herein again.
In other embodiments, to ensure security of data transmission and prevent security problems such as data leakage, cracking, monitoring, and the like, the server may further perform the following steps before merging a plurality of file fragments in a specified path into a large file based on the fragment identifier of the file fragment:
and under the condition that the client encrypts each file fragment and performs abstract calculation on the large file, decrypting the file fragment ciphertext received by each second thread to obtain the corresponding file fragment, and performing integrity check on the large file obtained by combination through the large file abstract received by the target second thread, wherein the target second thread is a second thread corresponding to the target first thread for transmitting the large file abstract.
In the embodiment of the invention, the second threads correspond to the first threads for uploading the file fragments one by one, so for the first threads for simultaneously transmitting the file fragments and the large file summaries, the corresponding second threads, namely the target second threads, can extract the large file summaries, and after the plurality of file fragments are combined into the large file based on the fragments, the integrity of the large file is checked through the large file summaries. For specific details, reference may be made to the parts disclosed in the foregoing embodiments of the method for transmitting a large file, and details are not described herein again.
In other embodiments, to deal with these emergencies and avoid unnecessary waste of resources and time, the server may receive, through each second thread, a file fragment uploaded by the corresponding first thread by using the breakpoint resume technique. For specific details, reference may be made to the parts disclosed in the foregoing embodiments of the method for transmitting a large file, and details are not described herein again.
Therefore, the invention can improve the transmission performance, the safety performance and the stability performance of the large file.
Based on the large file transmission method provided in the foregoing embodiment, an embodiment of the present invention provides an apparatus for executing the large file transmission method, where a schematic structural diagram of the apparatus is shown in fig. 6, and the apparatus includes:
the thread monitoring module 201 is configured to monitor a plurality of first threads of the client, where the first threads are used for transmitting file fragments of a large file.
A thread creation module 202 for creating a second thread corresponding to each first thread.
The second thread calling module 203 is configured to call each second thread to receive, through each second thread, the file fragment uploaded by the corresponding first thread.
The file merging module 204 is configured to store the received multiple file fragments to a specified path; and merging a plurality of file fragments under the specified path into a large file based on the fragment identification of the file fragment.
Optionally, the file merging module 204 is further configured to:
and under the condition that the client encrypts each file fragment and performs abstract calculation on the large file, decrypting the file fragment ciphertext received by each second thread to obtain the corresponding file fragment, and performing integrity check on the large file obtained by combination through the large file abstract received by the target second thread, wherein the target second thread is a second thread corresponding to the target first thread for transmitting the large file abstract.
Optionally, the second thread invoking module 203 is configured to receive, by each second thread, the file fragment uploaded by the corresponding first thread, and specifically configured to:
and receiving the file fragments uploaded by the corresponding first thread by adopting a breakpoint continuous transmission technology through each second thread.
In the embodiment of the present invention, the detailed functions of the modules may refer to the disclosure of the method embodiment, and are not described herein again.
The method, the device and the system for transmitting the large file provided by the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include or include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A large file transmission method is applied to a client, and comprises the following steps:
acquiring a large file to be transmitted;
fragmenting the large file to obtain a plurality of file fragments, wherein each file fragment has a fragment identifier capable of representing the position of the file fragment in the large file;
the method comprises the steps of calling a plurality of first threads, distributing file fragments to be transmitted for each first thread, uploading the distributed file fragments to a server through each first thread, and enabling the server to merge the plurality of file fragments into the large file based on fragment identification.
2. The method of claim 1, further comprising:
and performing type verification on the large file, and after the type verification is passed, executing fragmentation on the large file to obtain a plurality of file fragments.
3. The method of claim 1, further comprising:
encrypting each file fragment to obtain a corresponding file fragment ciphertext, and performing summary calculation on the large file to obtain a large file summary;
correspondingly, the uploading the allocated file fragments to the server by each first thread includes:
uploading file fragment ciphertexts corresponding to the distributed file fragments to a server through each first thread, determining a target first thread used for transmitting the large file abstract in the first threads, uploading the large file abstract to the server through the target first thread while uploading the file fragment ciphertexts, and enabling the server to obtain the file fragments through deciphering the file fragment ciphertexts and carrying out integrity check on the large file obtained through combination through the large file abstract.
4. The method of claim 1, wherein uploading the allocated file fragments to the server by each first thread comprises:
and uploading the distributed file fragments to the server side through each first thread by adopting a breakpoint continuous transmission technology.
5. A large file transfer apparatus, comprising:
the file acquisition module is used for acquiring a large file to be transmitted;
the file fragmentation module is used for fragmenting the large file to obtain a plurality of file fragments, and each file fragment is provided with a fragment identifier capable of representing the position of the file fragment in the large file;
the first thread calling module is used for calling a plurality of first threads and distributing the file fragments to be transmitted for each first thread so as to upload the distributed file fragments to the server side through each first thread, and the server side can combine the file fragments into the large file based on the fragment identification.
6. A large file transmission method is applied to a server side and comprises the following steps:
monitoring a plurality of first threads of file fragments of a large file for transmission of a client;
creating a second thread corresponding to each first thread;
calling each second thread to receive the file fragments uploaded by the corresponding first thread through each second thread;
storing the received multiple file fragments to a specified path;
and merging the plurality of file fragments under the specified path into the large file based on the fragment identification of the file fragment.
7. The method of claim 6, wherein before the file-slice-based slice identifier merges the plurality of file slices in the specified path into a large file, the method further comprises:
and under the condition that the client encrypts each file fragment and performs digest calculation on the large file, decrypting the file fragment ciphertext received by each second thread to obtain the corresponding file fragment, and performing integrity check on the large file obtained by combination through the large file digest received by a target second thread, wherein the target second thread is a second thread corresponding to a target first thread for transmitting the large file digest.
8. The method of claim 6, wherein receiving, by each second thread, the file slice uploaded by the corresponding first thread comprises:
and receiving the file fragments uploaded by the corresponding first thread by adopting a breakpoint continuous transmission technology through each second thread.
9. A large file transfer apparatus, comprising:
the thread monitoring module is used for monitoring a plurality of first threads for transmitting file fragments of the large file at the client;
a thread creating module for creating a second thread corresponding to each first thread;
the second thread calling module is used for calling each second thread so as to receive the file fragments uploaded by the corresponding first thread through each second thread;
the file merging module is used for storing the received multiple file fragments to a specified path; and merging the plurality of file fragments under the specified path into the large file based on the fragment identification of the file fragment.
10. A large file transfer system, the system comprising: a server and at least one client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011337777.9A CN112615899A (en) | 2020-11-25 | 2020-11-25 | Large file transmission method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011337777.9A CN112615899A (en) | 2020-11-25 | 2020-11-25 | Large file transmission method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112615899A true CN112615899A (en) | 2021-04-06 |
Family
ID=75225464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011337777.9A Pending CN112615899A (en) | 2020-11-25 | 2020-11-25 | Large file transmission method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112615899A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113568884A (en) * | 2021-09-26 | 2021-10-29 | 武汉四通信息服务有限公司 | File management method and device, electronic equipment and storage medium |
| CN114338122A (en) * | 2021-12-23 | 2022-04-12 | 广西东信数建信息科技有限公司 | Efficient and safe mass file transmission method |
| CN114448975A (en) * | 2022-03-04 | 2022-05-06 | 中国建设银行股份有限公司 | File transmission method, device, equipment and medium |
| CN114567506A (en) * | 2022-03-09 | 2022-05-31 | 平安科技（深圳）有限公司 | File uploading method and device, computer equipment and storage medium |
| CN115037741A (en) * | 2022-08-11 | 2022-09-09 | 中国长江三峡集团有限公司 | File transmission method and device |
| CN115086298A (en) * | 2022-05-30 | 2022-09-20 | 中银金融科技有限公司 | File transmission method and device |
| CN115150133A (en) * | 2022-06-20 | 2022-10-04 | 浙江大华技术股份有限公司 | Data transmission method and device and electronic equipment |
| WO2023103190A1 (en) * | 2021-12-06 | 2023-06-15 | 福建亿榕信息技术有限公司 | Multi-level linkage transparent sample model sharing apparatus for artificial intelligence platform |
| CN116383896A (en) * | 2023-06-07 | 2023-07-04 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
-
2020
- 2020-11-25 CN CN202011337777.9A patent/CN112615899A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113568884A (en) * | 2021-09-26 | 2021-10-29 | 武汉四通信息服务有限公司 | File management method and device, electronic equipment and storage medium |
| WO2023103190A1 (en) * | 2021-12-06 | 2023-06-15 | 福建亿榕信息技术有限公司 | Multi-level linkage transparent sample model sharing apparatus for artificial intelligence platform |
| CN114338122A (en) * | 2021-12-23 | 2022-04-12 | 广西东信数建信息科技有限公司 | Efficient and safe mass file transmission method |
| CN114448975A (en) * | 2022-03-04 | 2022-05-06 | 中国建设银行股份有限公司 | File transmission method, device, equipment and medium |
| CN114567506A (en) * | 2022-03-09 | 2022-05-31 | 平安科技（深圳）有限公司 | File uploading method and device, computer equipment and storage medium |
| CN114567506B (en) * | 2022-03-09 | 2024-03-19 | 平安科技（深圳）有限公司 | File uploading method and device, computer equipment and storage medium |
| CN115086298A (en) * | 2022-05-30 | 2022-09-20 | 中银金融科技有限公司 | File transmission method and device |
| CN115150133A (en) * | 2022-06-20 | 2022-10-04 | 浙江大华技术股份有限公司 | Data transmission method and device and electronic equipment |
| CN115037741B (en) * | 2022-08-11 | 2022-11-15 | 中国长江三峡集团有限公司 | File transmission method and device |
| EP4322018A1 (en) * | 2022-08-11 | 2024-02-14 | China Three Gorges Corporation | A file transmitting method and apparatus |
| CN115037741A (en) * | 2022-08-11 | 2022-09-09 | 中国长江三峡集团有限公司 | File transmission method and device |
| CN116383896A (en) * | 2023-06-07 | 2023-07-04 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
| CN116383896B (en) * | 2023-06-07 | 2023-11-03 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112615899A (en) | Large file transmission method, device and system | |
| CN109428899A (en) | File security transfer management method and system | |
| CN106549933B (en) | Data transmission system and method of block chain | |
| US10764261B2 (en) | System and method for enabling a scalable public-key infrastructure on a smart grid network | |
| CN109194669B (en) | Data transmission method, device, equipment and medium of lightweight node | |
| CN114466015B (en) | Data storage system and method based on multi-cloud architecture | |
| CN102055580A (en) | Method for safely sending and receiving enterprise information in industrial internet and communication equipment | |
| US11671279B2 (en) | Determining a session key using session data | |
| CN110809026B (en) | File processing method and device, electronic equipment and storage medium | |
| US20120250865A1 (en) | Securely enabling access to information over a network across multiple protocols | |
| CN103747039B (en) | Digital film copy distribution system | |
| WO2023061060A1 (en) | Audio and video code stream scheduling method, system, medium and electronic apparatus | |
| CN101667999A (en) | Method and system for transmitting peer-to-peer broadcast stream, data signature device and client | |
| US9332405B2 (en) | Short message backup method, mobile terminal, and server | |
| CN113515766A (en) | File transmission method and device | |
| CN111416992A (en) | File transcoding method and device | |
| CN116566661A (en) | Method, device, electronic equipment and storage medium for data security transmission | |
| CN109660568B (en) | Method, equipment and system for realizing network talkback security mechanism based on SRTP | |
| CN111125426A (en) | Data storage and query method and device | |
| CN113765968A (en) | File transmission method, device and system | |
| CN111355793A (en) | Large file uploading method based on HTML5 and Python | |
| CN113472722A (en) | Data transmission method, storage medium, electronic device and automatic ticket selling and checking system | |
| US20130024543A1 (en) | Methods for generating multiple responses to a single request message and devices thereof | |
| CN111506913B (en) | Audio encryption method and device, storage medium and electronic device | |
| CN112751842A (en) | High-performance instant messaging method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211108 Address after: 100192 room 710, research building 15, Xiaoying East Road, Qinghe, Haidian District, Beijing Applicant after: BEIJING CHINA POWER INFORMATION TECHNOLOGY Co.,Ltd. Address before: 100192 room 710, research building 15, Xiaoying East Road, Qinghe, Haidian District, Beijing Applicant before: BEIJING CHINA POWER INFORMATION TECHNOLOGY Co.,Ltd. Applicant before: STATE GRID INFORMATION & TELECOMMUNICATION GROUP Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |