CN108933805A - A kind of document transmission method and system - Google Patents

A kind of document transmission method and system Download PDF

Info

Publication number
CN108933805A
CN108933805A CN201710386068.1A CN201710386068A CN108933805A CN 108933805 A CN108933805 A CN 108933805A CN 201710386068 A CN201710386068 A CN 201710386068A CN 108933805 A CN108933805 A CN 108933805A
Authority
CN
China
Prior art keywords
file
slice
specified
byte
stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710386068.1A
Other languages
Chinese (zh)
Inventor
李春祝
张文明
陈少杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Douyu Network Technology Co Ltd
Original Assignee
Wuhan Douyu Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Douyu Network Technology Co Ltd filed Critical Wuhan Douyu Network Technology Co Ltd
Priority to CN201710386068.1A priority Critical patent/CN108933805A/en
Publication of CN108933805A publication Critical patent/CN108933805A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

The present invention provides a kind of document transmission method and system.The method includes S101, send file upload request to server-side, include the file name and file total length of specified file in the file upload request;S102 obtains the server-side and is directed to the file unique identification that the file upload request returns;S103 is identified the specified file using the file unique identification, uploads multiple slice file streams of the specified file to the server-side respectively using multiple sub threads;The multiple file slice stream is that the specified file is split to form by specified slice size, and flows into line identifier to the multiple file slice using the file unique identification.The present invention utilizes multi-threading parallel process, and speed is fast, supports super big file uploading, client and server-side, which are combined, realizes slicing treatment mechanism, to solve the problems, such as big file uploading.

Description

A kind of document transmission method and system
Technical field
The present invention relates to field of computer technology, more particularly, to a kind of document transmission method and system.
Background technique
Currently, general enterprises all use File Transfer Protocol (FTP) tool to upload big file.File Transfer Protocol (FTP) It is configured to an agreement that is cross-platform, simple and being easily achieved.File Transfer Protocol (FTP) has a very long evolution History is one of most important application on internet, but even to this day, has but been gone from bad to worse.
File Transfer Protocol (FTP) behaviour the shortcomings that denouncing includes:The first, FTP configuration is comparatively laborious.The second, data pass The unreasonable content for not considering file itself of defeated mode is simply unreasonable using ascii mode transmission data.Third, work It is designed as mode and most allows people is mysterious to be in unreasonable File Transfer Protocol (FTP), client can listen to server end.The Four, cryptosecurity strategy is not perfect in ftp client and server end, and data are transmitted in the form of plaintext, any pair of communication road Routing on diameter has the people of control ability, your password and data can be obtained by sniff.We can of course use SSL encapsulates FTP, but FTP is carried out data transmission by establishing repeatedly link, we even protect cryptosecurity, It is difficult to protect the safety of data transmission.5th, File Transfer Protocol inefficiency.A file is retrieved from ftp server, includes Complicated exchange handshake steps transmit a file, and FTP needs reciprocal 10 times, and HTTP only needs 2 times!If transmitted multiple File, FTP can be omitted the step of sending username and password, and fixed socket (Socket) then can be used in HTTP, File is transmitted in identical TCP connection.
Although when having been subjected to now, it is one in conclusion File Transfer Protocol (FTP) is once much-vaunted It is neither dangerous, it is also not unfriendly, and the agreement of inefficiency.
Under the conditions of current application, user widely can upload the relevant data of business to file using smart phone or PC Server, uplink data include document, picture, compressed file packet, video file of large capacity etc..Accomplish the client of millions Upper transmitting file, FTP much can not meet existing demand, be badly in need of a kind of new firms grade super large file uploading method simultaneously.
Summary of the invention
The present invention provide a kind of document transmission method for overcoming the above problem or at least being partially solved the above problem and System.
First embodiment according to the present invention provides a kind of document transmission method, is applied to client, including:
S101 sends file upload request to server-side, includes the filename of specified file in the file upload request Claim and file total length;
S102 obtains the server-side and is directed to the file unique identification that the file upload request returns;
S103 is identified the specified file using the file unique identification, using multiple sub threads respectively to The server-side uploads multiple slice file streams of the specified file;The multiple file slice stream is that the specified file is pressed Specified slice size is split to form, and flows into line identifier to the multiple file slice using the file unique identification.
Second embodiment according to the present invention provides a kind of document transmission method, is applied to objective server-side, including:
S201 receives the file upload request that client is sent, and includes file name and text in the file upload request Part total length;
S202 generates the file unique identification for being directed to the file upload request, and returns to client;
S203 successively receives multiple slice file streams that the client is sent using the different threads in thread pool, will The multiple slice file stream is merged into a file;Wherein, one file is marked by the file unique identification Know.
Third embodiment according to the present invention provides a kind of client, including:
Request module is sent, for sending file upload request to the server-side, includes in the file upload request The file name and file total length of specified file;
Mark module is obtained, is uniquely marked for obtaining the file that the server-side is returned for the file upload request Know;And
File module is uploaded, for being identified using the file unique identification to the specified file, utilization is multiple Sub thread uploads multiple slice file streams of the specified file to the server-side respectively;The multiple file slice stream is institute It states specified file to be split to form by specified slice size, and is identified using the file unique identification.
The present invention proposes a kind of document transmission method and system, and for super big file uploading, client will upload big first File is sliced into many small slicing blocks, then creates multiple sub threads and the slice file of client is streamed to server-side;Clothes Business end it is responsible for every small block file stream that same file is written in sequence.The present invention utilizes multi-threading parallel process, and speed is fast, Support super big file uploading, client and server-side, which are combined, realizes slicing treatment mechanism, to solve the problems, such as big file uploading.
Detailed description of the invention
Fig. 1 is a kind of document transmission method client flow chart of the embodiment of the present invention;
Fig. 2 is a kind of document transmission method server-side flow chart of the embodiment of the present invention;
Fig. 3 is a kind of structural block diagram of document transmission device of the embodiment of the present invention.
Specific embodiment
With reference to the accompanying drawings and examples, specific embodiments of the present invention will be described in further detail.Implement below Example is not intended to limit the scope of the invention for illustrating the present invention.
As shown in Figure 1, first embodiment of the invention provides a kind of document transmission method, it is applied to client, including:
S101 sends file upload request to server-side, includes the filename of specified file in the file upload request Claim and file total length;
S102 obtains the server-side and is directed to the file unique identification that the file upload request returns;
S103 is identified the specified file using the file unique identification, using multiple sub threads respectively to The server-side uploads multiple slice file streams of the specified file;The multiple file slice stream is that the specified file is pressed Specified slice size is split to form, and flows into line identifier to the multiple file slice using the file unique identification.
A kind of document transmission method described in the embodiment of the present invention, can be used for the transmission of enterprise-level super large file, and this method is Client approach.
Super large file described in the embodiment of the present invention, is a kind of opposite concept, and big file has for small documents Biggish data volume, super large file have bigger data volume relative to big file.File transmission dependent on computer hardware and Network configuration, different computer hardware develop the big file in epoch and small documents be also it is different, in a developing stage Big file, may be exactly small documents in next developing stage.In general, in certain computer hardware and network condition Under, big file transmission needs to consume more resources, needs more transmission times.
Enterprise-level super large file, refers to for personal document described in the embodiment of the present invention, in enterprise production process File data volume it is very big, when data volume greatly to a certain extent after will to data transmit be file transmission cause great shadow It rings, it is therefore necessary to the transmission problem of enterprise-level super large file is solved by a kind of method.
For super big file uploading, client will upload big file first and be sliced into many small slicing blocks, then create more A sub thread is responsible for the slice file of client to be streamed to server-side using multiple sub threads.
What the present embodiment client uploaded is slice file stream, and a slice file stream is by a sub thread transmission, often The mutual independent operating of a slice sub thread, does not interfere with each other, does not influence the stability of main program.Multi-threading parallel process speed is fast, It by the big file of LAN testing 100,000,000, uploaded successfully within the used time 5 seconds, to support super big file uploading.
Wherein, the file upload request, for telling server-side client to need upper transmitting file, and incidentally upper transmitting file Some necessary informations, such as file name and file total length, it is of course possible to which other information is attached to according to concrete application situation.
Wherein, the file unique identification is generated by server-side, is identified for transmitting file on the client to file, It is also used for being identified file after server-side receives file.When a server-side corresponds to multiple client, different visitors Family end all transmitting file, the present embodiment can uniformly carry out resource allocation by server-side in server-side, and server-side passes through unification File identification realizes the management to upper transmitting file.
In one embodiment of the first embodiment, the S101 further comprises:
Specified safety detection mode is obtained, based on the specified safety detection mode in the file upload request Specified the safety detection mode and particular detection information is added;
Wherein, the specified safety detection mode is public upload, provides and download using downloading upload and setting key Any one of upload;
Wherein, when the specified safety detection mode is to provide to upload using downloading, the particular detection information is The unique identification of specified application;When the specified safety detection mode is that setting key downloads upload, the particular detection Information is specified key.
The present embodiment provides safety detection mechanism when a kind of upload of file.In general, client is sent out to server-side When sending file upload request, the security mechanism of safety inspection, the access of transmitting file can be carried out, upload trace, file upload Controlling mechanism etc..According to the different application of enterprises, corresponding application id and authentication tokens, file can be configured Verification Application Id and authentication tokens are needed when upload, if being impermissible for uploading using not registering;Also it is quick that filename is added Feel information checking, blacklist verification, specified application or upload verification of Tocken downloading etc..Three kinds of safety described in the present embodiment Detection mode is respectively:
1) public upload:File upload is not limited, and being not required to verification can upload.
2) upload of application downloading is provided:It needs to verify file upload, if meeting specified application, can upload, it is no It cannot then upload, the unique identification that parameter is application must be passed.
3) upload that setting code key is downloaded:It needs to verify file upload, if code key matches, can upload, otherwise not It can upload, must pass parameter is code key.
The present embodiment samples the safety detection that proprietary protocol carries out file upload, and concurrent processing performance can be improved, and is promoted The safety that file uploads.
In another embodiment of the first embodiment, multiple file slice streams are the finger described in the S103 Determine file by specified slice size be split to form including:
Step 1, the file name and file path for obtaining the specified file obtain the file overall length of the specified file Degree;
Step 2, it is based on the file total length and specified slice size, is obtained using file Slicing Algorithm described specified The slice information of file, including:
Maximum number of threads=slice numbers=file size/specified slice size;
The BOB(beginning of block) position of n-th slice information=(N-1) * specifies slice size.
Agllutination beam position=N* of n-th slice information specifies slice size, and if last block, agllutination beam position is institute State the last byte location of specified file;
Slice length=agllutination beam position-BOB(beginning of block) position;
Wherein, 1<=N<=slice numbers;
Step 3, the slice information based on the specified file obtains multiple slice file streams.
The present embodiment obtains the slice information for needing the file uploaded using file Slicing Algorithm, has slice information, just File can be sliced, be divided into multiple and different slice file stream;It is having the same that file stream is respectively sliced in the present embodiment Byte number can guarantee that client is that each uploads thread distribution byte number as identical as possible in this way, reach concurrency performance It optimizes.
In the present embodiment, method described by the step 1, step 2 and step 3, the opportunity executed is not necessarily described In step S103, can also before step S103, including before step S101, before step S102 and before step S103, As long as being handled before uploading to server-side, the present embodiment executes opportunity with no restriction to specific.
Further, step 1, step 2 and step 3 described in the present embodiment execute sequence with successive, but might not need It to execute together, can sequentially be inserted into the step in step S101, step S102 and step S103 arbitrary steps 1, the processing of one of step of step 2 and step 3.For example, the step 1 and step 2 are executed before step S101, The step 3 is executed in step S103;Can certainly before step S101 or step S102 by the step 1, step 2 and Step 3, which is carried out, to be finished;It certainly further include other embodiments, it is numerous to list herein.
Wherein, the specified slice size can be depending on specific application demand and device network environmental parameter, this Embodiment is not especially limited this.
In a specific example, the acquisition process for being sliced file stream is as follows:
1) the file total length for needing the file uploaded, such as file.length () in java are obtained.
2) slice size is specified, Thread Count is calculated according to slice size.Such as slice size is 10M, the file of 110M Can be sliced as 11 blocks, 11 is exactly number of threads.
File Slicing Algorithm is:
Thread_num_max=block_num=file_length/block_size;
Block_startpos=(N-1) * block_size;
Block_endpos=N*block_size;Or block_endpos=file_length;
Wherein, thread_num_max is the maximum thread that upper transmitting file needs, and block_num is number of sections, File_length is file total length, and block_size is specified slice length;N is integer, and 1<=N<=thread_ Num_max, block_startpos are BOB(beginning of block) position, and block_endpos is agllutination beam position.
Wherein, block_endpos=N*block_size is the agllutination beam position of last non-block, block_endpos =file_length is the agllutination beam position of last block.
In another embodiment of the first embodiment, the S103 further comprises:
S103.1 is based on the multiple slice file stream, distributes a sub thread for each slice file stream, utilizes institute It states file unique identification, the number of one sub thread and byte offsets and rower is flowed into each described slice file Know;Wherein, the byte offsets are the BOB(beginning of block) position in the slice information of each slice file stream;
S103.2, based on each slice file stream, using distribute to the sub thread of each slice file stream to The server-side sends each described slice file stream.
In the present embodiment, client's is distributed equally the task of upper transmitting file, and the specified file is sliced into finger After determining slice size (such as:Cut by 10M), first according to original document size, slice size, calculate the small documents being finally sliced Number N opens multiple threads according to Thread Scheduling Algorithms, distributes a sub thread for each slice file stream.
Each slice file stream has file unique identification identical with the specified file, carries out for server-side Identification.Meanwhile each slice file stream identifies the thread number oneself used, the byte relative to the specified file The length of the offset of starting point and upper transmitting file.
Multithreading upload has used customized Thread Scheduling Algorithms, customized Thread Scheduling Algorithms' packet in the present embodiment It includes:Poll creates sub thread, and slice the file information is called in memory, distributes resource for them;Newly generated slice the file information It is put into ready queue, until file slicing processes are fully completed.Multiple sub thread asynchronous execution file slicing actions, when being sliced At the state in modification queue.
Algorithm citing:If there is 10 channels, 20 vehicles, each channel can only exercise a vehicle every time, when vehicle sails out of Behind channel, channel is in idle condition, and dispatcher needs to distribute at this time a vehicle and comes in, and guarantees that each channel makes full use of, Channel is efficiently utilized, congested with cars is reduced.Dispatcher's poll distribution, until all vehicles sail out of channel.
The key condition of the customized Thread Scheduling Algorithms is:It is needed most when the number of files of completion is less than upper transmitting file When big Thread Count, circulation executes slicing action.Number of threads can be assigned it can assign number of threads equal to maximum and subtract and work Number of threads (increment generate subfile piece number-increment complete subfile piece Thread Count), a variable i is set, when The subfile piece number that variable i, which is less than, can assign number of threads and increment generates is less than the maximum thread that upper transmitting file needs When, it indicates i idle channel, i new slice file streams can be distributed.
Further include step 0 before the step 1 in another embodiment of the first embodiment, is based on the finger Determine file building file operation object, the file operation object is included at least as properties:
The maximum thread that file ID, file name, file path, file total length and upper transmitting file need;Wherein institute Stating file ID is the file unique identification obtained in the S102;
Further include step 21 after the step 2, based on the slice information of the specified file, constructs block_num Blocks of files operation object, and be added in the file operation object;The blocks of files operation object is included at least as properties:
Block number, file block length, BOB(beginning of block) position and agllutination beam position.
This gives the schemes that a kind of client handles the specified file and slice information.Client File slice information is calculated, according to the file unique identification sourceID that server-side returns, constructs file operation object FileInfo;Thread Scheduling Algorithms are called, scheduling generates new sub thread, constructs blocks of files operation object according to sourceID TaskInfo distributes thread for TaskInfo, and poll execution thread dispatching algorithm realizes that entire file uploads.
Wherein, file operation object FileInfo important attribute includes:File name, upload uniqueness file id, on The maximum thread that transmitting file, TaskInfos file slice block message, write-in file byte number, transmission rate, upper transmitting file need The configurable maximum number of threads in number, customer side, file total length, file upload state (0:Failure, 1:Successfully), it uploads People uploads IP address, the port port etc., wherein file upload state can be in file upload procedure according to practical shape State is modified.
Wherein, blocks of files operation object TaskInfo important attribute includes:Corresponding thread number, reading file size, BOB(beginning of block) position, agllutination beam position, uplink time, time-consuming, thread state (0:Running operation, 1:Over overload, 2:hold It is handling, 3:Error mistake), upload error message etc..
By a specific code example come supporting paper block operation object TaskInfo and file operation object The relationship of FileInfo:
In another embodiment of the first embodiment, the step 3 further comprises:
BOB(beginning of block) position and agllutination beam position based on slice information, read in the specified file and open from described piece Beginning position generates one slice file stream to whole bytes of agllutination beam position;Wherein, the reading of any byte is walked Suddenly include:
It inquires and whether there is any byte in current buffer;If it exists, then it is read from the current buffer Any byte;
If it does not exist, the current buffer navigates to the specific position where any byte, and by the spy The byte content for positioning the preset length for setting beginning is filled into the current buffer, deposits until inquiring current buffer again Any byte is read in any byte.
The present embodiment is the specific processing method that slice file stream is generated according to slice information.In order to solve traditional file Frequent progress I/O operation in data flow interception and lead to the file of the big low efficiency of expense, the present embodiment passes through step 31, step 32 Processing is optimized with step 33.
For example with java program, client transmits file stream, has repeatedly used the I/O behaviour of file Make, it is slower that RandomAccessFile efficiency is provided in java;It is then according to this embodiment, it can newly-increased BuffRandomAccessFile encapsulates class, promotes transmission speed in the way of the trivial reading of buffering;It is described BuffRandomAccessFile encapsulates the extension class that class is RandomAccessFile.
In the per thread passed on the client, using the seek function of BuffRandomAccessFile, reading is referred to The section first place that needle navigates to each section in original is set, and then reads specified slice size backward (i.e.:It is sliced block size), finally Corresponding slice file stream is written into file data, each slice file stream is transferred to server-side by the interface channel established.
The BuffRandomAccessFile encapsulation class is allowed to have by extending RandomAccessFile class Buffer improves I/O performance.The I/O efficiency of the file arbitrary access class RandomAccessFile of JAVA is lower, and the present invention is logical Analysis wherein reason is crossed, solution is proposed, gradually shows how to create the file arbitrary access for having buffer area read-write ability Class, and be optimized.Pass through the performance comparison with other file access classes, it was demonstrated that its practical value.
RandomAccessFile one byte of every read/write just needs to carry out an I/O operation to disk, and efficiency is relatively Slowly, Buffered I/O putStream one byte of every read/write, to operation data in BUF, just directly to memory Buf [] carry out read/write operation;Otherwise buf [] is filled from disk corresponding position, then directly the buf [] of memory read/ Write operation, most read/write operations are the operations to memory buf [], and efficiency is very fast.Memory access time unit is to receive Second grade (10E-9), disk address time unit is Millisecond (10E-3), equally operates primary expense, memory is faster than disk Million times.Theoretically it is contemplated that, even if up to ten thousand to internal memory operation secondary, the time of cost is also far fewer than to I/O's of disk Expense.The latter be by increase be located at memory BUF access, reduce the expense of magnetic disc i/o, improve access efficiency, certainly this Sample also increases the expense of the control section BUF.From the point of view of experiment, access efficiency improves 32 times.So decision pair RandomAccessFile is packaged, in addition buffer area reading mechanism.
Optimization principles:
Frequent sentence is called to need most optimization, and the effect optimized is most obvious.
When multinest logic judgment, the judgement that most probable occurs should be placed on outermost layer.
Reduce unnecessary NEW.
Class is encapsulated by BuffRandomAccessFile to optimize the seek function in RandomAccessFile, Including following two kinds:
Optimization one:This.bufstartpos=(pos<<bufbitlen)>>bufbitlen;
Optimization two:This.bufstartpos=pos&bufmask;//this.bufmask=~((long) this.bufsize-1);
The two efficiency is all better than original, but the latter is obviously more preferable, because the former needs shift operation twice, the latter only to need one Secondary logic and operation (bufmask can be obtained in advance).Both the above optimization is the citing of one of java program, thought Principle is equally applicable in other program languages, also within the protection scope of the present embodiment.
Specifically, a byte of file POS location to be read, the basic principle for reading buffer logic include:
Looking into buffer area whether there is?If so, the middle reading directly from buffer area, and return to character BYTE.
If the position where the POS no, will be repositioned onto buffer area and the BUFSIZE word near the position In the file content filling buffer area of section, previous step is returned.
Wherein, the BUFSIZE byte is the byte of the preset length, and specific length can be according to practical application request Depending on, the present embodiment is not construed as limiting this.
As shown in Fig. 2, second embodiment of the invention provides a kind of document transmission method, it is applied to objective server-side, including:
S201 receives the file upload request that client is sent, and includes file name and text in the file upload request Part total length;
S202 generates the file unique identification for being directed to the file upload request, and returns to client;
S203 successively receives multiple slice file streams that the client is sent using the different threads in thread pool, will The multiple slice file stream is merged into a file;Wherein, one file is marked by the file unique identification Know.
A kind of document transmission method of the present embodiment is server-side method, and method corresponding with the client equally may be used With the transmission for enterprise-level super large file.Server-side successively receives the slice file stream of client using thread pool mode, What is received is written same file per small block file stream in sequence.The biography that All Files are sliced is completed when uploading client Defeated, server-side is responsible for being assembled into complete file, that is, completes big file slice upload movement.
Server-side generates the file unique identification, is identified, is also used for file for transmitting file on the client Server-side is identified file after receiving file.When a server-side corresponds to multiple client, different clients is all Transmitting file, the present embodiment resource allocation can uniformly be carried out by server-side in server-side, server-side passes through unified files-designated Know the management realized to upper transmitting file.
In one embodiment of the second embodiment, the multiple slice file stream is merged described in the S203 Further comprise at a file:
The multiple slice file streams received are corresponded, multiple temporary files are written;Text is completed when listening to client When part uploads, based on the byte offsets in each slice file stream, the multiple temporary file is merged into one File;Or
Based on the file name and file total length in the file upload request, one file is created;Work as reception When being sliced stream to file, institute is written in the corresponding position of one file according to the byte offsets in file slice stream State the data of file slice stream.
In the present embodiment, server-side can also be corresponding with client, and temporary file operation is constructed after resolution request message Object FileInfo.Server-side obtains the file stream that client sends over, and can directly write a whole file or write It is then whole file merging temporary file at multiple temporary files.To be write as the mode of temporary file, temporary file is write Block flows to local file, writes temporary file rule rootpath/ Year/Month/Day/file ID/filename extension name;Work as client It completes file to upload, server end uses thread pool mode to merge temporary file for complete file.Server-side can be to client The download address url for the file that the client uploads is returned to, file, which uploads, to be completed.
In another embodiment of the second embodiment, the multiple slice file stream is merged described in the S203 Include at a file:The byte in slice file stream is first read, then file is written into the byte of reading;Wherein, for slice The read step of any byte in file stream includes:
It inquires and whether there is any byte in current buffer;If it exists, then it is read from the current buffer Any byte;
If it does not exist, the current buffer navigates to the specific position where any byte, and by the spy The byte content for positioning the preset length for setting beginning is filled into the current buffer, deposits until inquiring current buffer again Any byte is read in any byte.
In the present embodiment, file is written in received slice file stream by server-side, the reading buffer principle phase with client Together.Either server-side writes a whole file after receiving slice file stream or is write as multiple temporary files, requires Using this writing buffer principle.Writing buffer principle actually write-after-read described in the present embodiment is substantially still to read buffering Area's principle.
Similar with client approach, server-side opens thread after receiving client slice file stream, utilizes The seek function of BuffRandomAccessFile (extension class of RandomAccessFile), navigates to file for reading pointer BOB(beginning of block) position, specified size is written (i.e. backward:It is sliced block size), corresponding slice file is finally written.(according to one The writing mode of a entirety file, it is also necessary to utilize the length of setLength file setting blocks of files.)
The BuffRandomAccessFile class and its seek optimization processing are identical as client, please refer to institute of the present invention The associated description of first embodiment is stated, details are not described herein again.
In another embodiment of the second embodiment, the S202 further comprises:
The safety detection mode and particular detection information specified based on the file upload request carry out safety detection, work as peace Full inspection survey to the client sends the file unique identification after passing through.
In the present embodiment, server-side carries out safety detection according to the particular content of the file upload request of client, please join The associated description of first embodiment of the present invention is examined, details are not described herein again.
Third embodiment of the invention provides a kind of client, including:
Request module is sent, for sending file upload request to the server-side, includes in the file upload request The file name and file total length of specified file;
Mark module is obtained, is uniquely marked for obtaining the file that the server-side is returned for the file upload request Know;And
File module is uploaded, for being identified using the file unique identification to the specified file, utilization is multiple Sub thread uploads multiple slice file streams of the specified file to the server-side respectively;The multiple file slice stream is institute It states specified file to be split to form by specified slice size, and is identified using the file unique identification.
Client described in the present embodiment for realizing client approach described in first embodiment of the invention and its it is all can The embodiment method of choosing specifically describes the associated description for please referring to first embodiment of the present invention, and details are not described herein again.
Fourth embodiment of the invention provides a kind of server-side, including:
Request module is received, for receiving the file upload request of client transmission, includes in the file upload request File name and file total length;
Mark module is sent, for generating the file unique identification for being directed to the file upload request, and returns to client End;And
File module is received, for successively receiving multiple the cutting of the client transmission using the different threads in thread pool The multiple slice file stream is merged into a file by piece file stream;Wherein, one file is unique by the file Mark is identified.
Server-side described in the present embodiment for realizing server-side method described in second embodiment of the invention and its it is all can The embodiment method of choosing specifically describes the associated description for please referring to second embodiment of the present invention, and details are not described herein again.
Fifth embodiment of the invention provides a kind of document transmission system, including client described in third embodiment of the invention And its server-side described in any optional embodiment and fourth embodiment of the invention and its any optional embodiment.
Fig. 3 shows a kind of structural block diagram of document transmission device of the embodiment of the present invention.
Reference Fig. 3, the equipment, including:Processor (processor) 601, memory (memory) 602 and bus 603;
Wherein, the processor 601 and memory 602 complete mutual communication by the bus 603.
Referring to Fig. 3, according to a sixth embodiment of the present invention, the processor 601 is used to call the journey in the memory 602 Sequence instruction, to execute method provided by above-mentioned each method embodiment, for example including:Establish the communication connection with server-side, base File upload request is sent to the server-side in the specified file for needing to upload, includes filename in the file upload request Claim and file total length;It obtains the server-side and is directed to the file unique identification that the file upload request returns;Using described File unique identification is identified the specified file, is uploaded respectively to the server-side using multiple sub threads described specified Multiple slice file streams of file.
According to a seventh embodiment of the present invention, a kind of computer program product is disclosed, and the computer program product includes depositing The computer program in non-transient computer readable storage medium is stored up, the computer program includes program instruction, when described When program instruction is computer-executed, computer is able to carry out method provided by above-mentioned each method embodiment, for example including:It builds The vertical communication connection with server-side sends file upload request to the server-side based on the specified file for needing to upload, described It include file name and file total length in file upload request;The server-side is obtained to return for the file upload request File unique identification;The specified file is identified using the file unique identification, is distinguished using multiple sub threads Multiple slice file streams of the specified file are uploaded to the server-side.
According to a seventh embodiment of the present invention, a kind of non-transient computer readable storage medium, the non-transient calculating are provided Machine readable storage medium storing program for executing stores computer instruction, and the computer instruction makes the computer execute above-mentioned each method embodiment institute The method of offer, for example including:Establish with the communication connection of server-side, based on the specified file for needing to upload to the server-side File upload request is sent, includes file name and file total length in the file upload request;Obtain the server-side needle The file unique identification that the file upload request is returned;The specified file is marked using the file unique identification Know, uploads multiple slice file streams of the specified file to the server-side respectively using multiple sub threads.
Referring to Fig. 3, according to a eighth embodiment of the present invention, the processor 601 is used to call the journey in the memory 602 Sequence instruction, to execute method provided by above-mentioned each method embodiment, for example including:The file upload that client is sent is received to ask It asks, includes file name and file total length in the file upload request;Generate the file for being directed to the file upload request Unique identification, and return to client;Multiple the cutting of the client transmission is successively received using the different threads in thread pool The multiple slice file stream is merged into a file by piece file stream;Wherein, one file is unique by the file Mark is identified.
According to a ninth embodiment of the present invention, a kind of computer program product is disclosed, and the computer program product includes depositing The computer program in non-transient computer readable storage medium is stored up, the computer program includes program instruction, when described When program instruction is computer-executed, computer is able to carry out method provided by above-mentioned each method embodiment, for example including:It connects The file upload request that client is sent is received, includes file name and file total length in the file upload request;Generate needle To the file unique identification of the file upload request, and return to client;It is successively connect using the different threads in thread pool Multiple slice file streams that the client is sent are received, the multiple slice file stream is merged into a file;Wherein, described One file is identified by the file unique identification.
With regard to embodiment according to the present invention, provides a kind of non-transient computer readable storage medium, the non-transient calculating Machine readable storage medium storing program for executing stores computer instruction, and the computer instruction makes the computer execute above-mentioned each method embodiment institute The method of offer, for example including:The file upload request that client is sent is received, includes filename in the file upload request Claim and file total length;The file unique identification for being directed to the file upload request is generated, and returns to client;Utilize thread Different threads in pond successively receive multiple slice file streams that the client is sent, and the multiple slice file stream is merged At a file;Wherein, one file is identified by the file unique identification.
Those of ordinary skill in the art will appreciate that:Realize that all or part of the steps of above method embodiment can pass through The relevant hardware of program instruction is completed, and program above-mentioned can be stored in a computer readable storage medium, the program When being executed, step including the steps of the foregoing method embodiments is executed;And storage medium above-mentioned includes:ROM, RAM, magnetic disk or light The various media that can store program code such as disk.
The embodiments such as document transmission device described above are only schematical, wherein described say as separation unit Bright unit may or may not be physically separated, and component shown as a unit can be or can not also It is physical unit, it can it is in one place, or may be distributed over multiple network units.It can be according to actual need Some or all of the modules therein is selected to achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying Out in the case where creative labor, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should Computer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingers It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation Method described in certain parts of example or embodiment.
In conclusion the present invention provides a kind of enterprise-level super large document transmission method, client, server-side, system, setting Standby and computer readable storage medium supports high concurrent, high-performance, high reliability.Pass through multiple check mechanism, it is ensured that data pass Defeated integrality and correctness, and can realize continual data transmission, speed is fast and stablizes safety.Suitable for Java platform, Ios platform, Android platform allow mobile phone (IOS/Android) application and the application of JAVA to realize the file site collection of plug-in type At convenient, no code is invasive.
Finally, the present processes are only preferable embodiment, it is not intended to limit the scope of the present invention.It is all Within the spirit and principles in the present invention, any modification, equivalent replacement, improvement and so on should be included in protection of the invention Within the scope of.

Claims (10)

1. a kind of document transmission method is applied to client, which is characterized in that including:
S101, sends file upload request to server-side, include in the file upload request specified file file name and File total length;
S102 obtains the server-side and is directed to the file unique identification that the file upload request returns;
S103 is identified the specified file using the file unique identification, using multiple sub threads respectively to described Server-side uploads multiple slice file streams of the specified file;The multiple file slice stream is the specified file by specified Slice size is split to form, and flows into line identifier to the multiple file slice using the file unique identification.
2. the method as described in claim 1, which is characterized in that the S101 further comprises:
Specified safety detection mode is obtained, is added in the file upload request based on the specified safety detection mode Specified the safety detection mode and particular detection information;
Wherein, the specified safety detection mode is public upload, provides and download and upload using downloading upload and setting key Any one of;
Wherein, when the specified safety detection mode is to provide to upload using downloading, the particular detection information is specified The unique identification of application;When the specified safety detection mode is that setting key downloads upload, the particular detection information To specify key.
3. the method as described in claim 1, which is characterized in that multiple file slice streams described in the S103 are described specified File by specified slice size be split to form including:
Step 1, the file name and file path for obtaining the specified file obtain the file total length of the specified file;
Step 2, it is based on the file total length and specified slice size, obtains the specified file using file Slicing Algorithm Slice information, including:
Maximum number of threads=slice numbers=file size/specified slice size;
The BOB(beginning of block) position of n-th slice information=(N-1) * specifies slice size.
Agllutination beam position=N* of n-th slice information specifies slice size, and if last block, agllutination beam position is the finger Determine the last byte location of file;
Slice length=agllutination beam position-BOB(beginning of block) position;
Wherein, 1<=N<=slice numbers;
Step 3, the slice information based on the specified file obtains multiple slice file streams.
4. method as claimed in claim 3, which is characterized in that the S103 further comprises:
S103.1 is based on the multiple slice file stream, distributes a sub thread for each slice file stream, utilizes the text Part unique identification, the number of one sub thread and byte offsets flow into line identifier to each described slice file;Its In, the byte offsets are the BOB(beginning of block) position in the slice information of each slice file stream;
S103.2, based on each slice file stream, using distributing to the sub thread of each slice file stream to described Server-side sends each described slice file stream.
5. method as claimed in claim 3, which is characterized in that the step 3 further comprises:
BOB(beginning of block) position and agllutination beam position based on slice information, read in the specified file from the BOB(beginning of block) position Whole bytes of agllutination beam position are set, one slice file stream is generated;Wherein, for the read step packet of any byte It includes:
It inquires and whether there is any byte in current buffer;If it exists, then described in being read from the current buffer Any byte;
If it does not exist, the current buffer navigates to the specific position where any byte, and by the certain bits The byte content for setting the preset length of beginning is filled into the current buffer, until inquiring current buffer again, there are institutes It states any byte and reads any byte.
6. a kind of document transmission method is applied to server-side, which is characterized in that including:
S201, receives the file upload request that client is sent, and includes that file name and file are total in the file upload request Length;
S202 generates the file unique identification for being directed to the file upload request, and returns to client;
S203 successively receives multiple slice file streams that the client is sent using the different threads in thread pool, will be described Multiple slice file streams are merged into a file;Wherein, one file is identified by the file unique identification.
7. method as claimed in claim 6, which is characterized in that merge the multiple slice file stream described in the S203 Further comprise at a file:
The multiple slice file streams received are corresponded, multiple temporary files are written;It is completed on file when listening to client When biography, based on the byte offsets in each slice file stream, the multiple temporary file is merged into one file; Or
Based on the file name and file total length in the file upload request, one file is created;When receiving text When part slice stream, the text is written in the corresponding position of one file according to the byte offsets in file slice stream The data of part slice stream.
8. method as claimed in claim 6, which is characterized in that merge the multiple slice file stream described in the S203 Include at a file:The byte in slice file stream is first read, then file is written into the byte of reading;Wherein, for slice The read step of any byte in file stream includes:
It inquires and whether there is any byte in current buffer;If it exists, then described in being read from the current buffer Any byte;
If it does not exist, the current buffer navigates to the specific position where any byte, and by the certain bits The byte content for setting the preset length of beginning is filled into the current buffer, until inquiring current buffer again, there are institutes It states any byte and reads any byte.
9. method as claimed in claim 6, which is characterized in that the S202 further comprises:
The safety detection mode and particular detection information specified based on the file upload request carry out safety detection, when safety is examined Survey to the client sends the file unique identification after passing through.
10. a kind of client, which is characterized in that including:
Request module is sent, includes specified in the file upload request for sending file upload request to the server-side The file name and file total length of file;
Mark module is obtained, is directed to the file unique identification that the file upload request returns for obtaining the server-side;With And
It uploads file module and utilizes multiple sub-lines for being identified using the file unique identification to the specified file Journey uploads multiple slice file streams of the specified file to the server-side respectively;The multiple file slice stream is the finger Determine file to be split to form by specified slice size, and is identified using the file unique identification.
CN201710386068.1A 2017-05-26 2017-05-26 A kind of document transmission method and system Pending CN108933805A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710386068.1A CN108933805A (en) 2017-05-26 2017-05-26 A kind of document transmission method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710386068.1A CN108933805A (en) 2017-05-26 2017-05-26 A kind of document transmission method and system

Publications (1)

Publication Number Publication Date
CN108933805A true CN108933805A (en) 2018-12-04

Family

ID=64451308

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710386068.1A Pending CN108933805A (en) 2017-05-26 2017-05-26 A kind of document transmission method and system

Country Status (1)

Country Link
CN (1) CN108933805A (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109947613A (en) * 2019-03-26 2019-06-28 北京百度网讯科技有限公司 File read test method and apparatus
CN109981729A (en) * 2019-02-14 2019-07-05 视联动力信息技术股份有限公司 Document handling method, device, electronic equipment and computer readable storage medium
CN110505298A (en) * 2019-08-23 2019-11-26 四川长虹电器股份有限公司 Optimization system and its method based on transmitting file on linux system
CN111125017A (en) * 2019-12-26 2020-05-08 深圳前海环融联易信息科技服务有限公司 Data storage method and device based on tree structure and computer equipment
CN111193785A (en) * 2019-12-20 2020-05-22 北京淇瑀信息科技有限公司 File cutting transmission method and device and electronic equipment
CN111338780A (en) * 2020-02-28 2020-06-26 中国工商银行股份有限公司 Transmission method and device for concurrent files
CN111404990A (en) * 2020-02-14 2020-07-10 Oppo(重庆)智能科技有限公司 File transmission method, device, client and storage medium
CN111767331A (en) * 2020-06-17 2020-10-13 北京思特奇信息技术股份有限公司 Multitask visual activation method of million-magnitude data in relational database
CN112073386A (en) * 2020-08-18 2020-12-11 浙江鸿城科技有限责任公司 Window-free randomly-selectable file transmission method based on UDP (user Datagram protocol)
CN112637341A (en) * 2020-12-22 2021-04-09 平安银行股份有限公司 File uploading method and device, electronic equipment and storage medium
CN112738157A (en) * 2020-12-11 2021-04-30 苏州浪潮智能科技有限公司 File uploading method, system and equipment
CN113542264A (en) * 2021-07-13 2021-10-22 杭州安恒信息技术股份有限公司 File transmission control method, device, equipment and readable storage medium
CN113992938A (en) * 2021-10-22 2022-01-28 兰州乐智教育科技有限责任公司 Video uploading method and device, electronic equipment and computer readable storage medium
CN114172898A (en) * 2022-02-10 2022-03-11 中国人民解放军海军工程大学 Large file remote transmission method
CN114584556A (en) * 2022-03-14 2022-06-03 中国工商银行股份有限公司 File transmission method and device
CN114598697A (en) * 2022-03-29 2022-06-07 浪潮云信息技术股份公司 Method and device for transmitting file and computer readable storage medium
CN114630192A (en) * 2022-03-16 2022-06-14 深圳法政信息技术有限公司 Video evidence storing method

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101656751A (en) * 2008-08-18 2010-02-24 北京数码大方科技有限公司 Method and system for accelerating file uploading and downloading
CN102137278A (en) * 2010-12-30 2011-07-27 广州市动景计算机科技有限公司 System and method for broadcasting and distributing streaming media based on mobile terminal
CN102355426A (en) * 2011-06-30 2012-02-15 北京神州泰岳软件股份有限公司 Method for transmitting off-line file and system
CN102833294A (en) * 2011-06-17 2012-12-19 阿里巴巴集团控股有限公司 File processing method and system based on cloud storage, and server cluster system
US8473585B1 (en) * 2012-06-26 2013-06-25 Citrix Systems, Inc. Multi-threaded optimization for data upload
CN103414959A (en) * 2013-07-15 2013-11-27 北京奇虎科技有限公司 Method and device for accelerating online video playing
CN104361057A (en) * 2014-10-31 2015-02-18 无锡天脉聚源传媒科技有限公司 Method and device for uploading multimedia file
CN104679830A (en) * 2015-01-30 2015-06-03 乐视网信息技术(北京)股份有限公司 File processing method and device
CN105472035A (en) * 2016-01-04 2016-04-06 百度在线网络技术(北京)有限公司 File transmission method and device
CN106790709A (en) * 2017-02-28 2017-05-31 烽火通信科技股份有限公司 Ftp file transmission method and ftp client based on asynchronization read-write

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101656751A (en) * 2008-08-18 2010-02-24 北京数码大方科技有限公司 Method and system for accelerating file uploading and downloading
CN102137278A (en) * 2010-12-30 2011-07-27 广州市动景计算机科技有限公司 System and method for broadcasting and distributing streaming media based on mobile terminal
CN102833294A (en) * 2011-06-17 2012-12-19 阿里巴巴集团控股有限公司 File processing method and system based on cloud storage, and server cluster system
CN102355426A (en) * 2011-06-30 2012-02-15 北京神州泰岳软件股份有限公司 Method for transmitting off-line file and system
US8473585B1 (en) * 2012-06-26 2013-06-25 Citrix Systems, Inc. Multi-threaded optimization for data upload
CN103414959A (en) * 2013-07-15 2013-11-27 北京奇虎科技有限公司 Method and device for accelerating online video playing
CN104361057A (en) * 2014-10-31 2015-02-18 无锡天脉聚源传媒科技有限公司 Method and device for uploading multimedia file
CN104679830A (en) * 2015-01-30 2015-06-03 乐视网信息技术(北京)股份有限公司 File processing method and device
CN105472035A (en) * 2016-01-04 2016-04-06 百度在线网络技术(北京)有限公司 File transmission method and device
CN106790709A (en) * 2017-02-28 2017-05-31 烽火通信科技股份有限公司 Ftp file transmission method and ftp client based on asynchronization read-write

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109981729A (en) * 2019-02-14 2019-07-05 视联动力信息技术股份有限公司 Document handling method, device, electronic equipment and computer readable storage medium
CN109947613A (en) * 2019-03-26 2019-06-28 北京百度网讯科技有限公司 File read test method and apparatus
CN110505298A (en) * 2019-08-23 2019-11-26 四川长虹电器股份有限公司 Optimization system and its method based on transmitting file on linux system
CN111193785A (en) * 2019-12-20 2020-05-22 北京淇瑀信息科技有限公司 File cutting transmission method and device and electronic equipment
CN111125017A (en) * 2019-12-26 2020-05-08 深圳前海环融联易信息科技服务有限公司 Data storage method and device based on tree structure and computer equipment
CN111125017B (en) * 2019-12-26 2023-10-13 深圳前海环融联易信息科技服务有限公司 Data storage method and device based on tree structure and computer equipment
CN111404990A (en) * 2020-02-14 2020-07-10 Oppo(重庆)智能科技有限公司 File transmission method, device, client and storage medium
CN111404990B (en) * 2020-02-14 2023-06-02 Oppo(重庆)智能科技有限公司 File transmission method, device, client and storage medium
CN111338780B (en) * 2020-02-28 2023-03-31 中国工商银行股份有限公司 Transmission method and device for concurrent files
CN111338780A (en) * 2020-02-28 2020-06-26 中国工商银行股份有限公司 Transmission method and device for concurrent files
CN111767331A (en) * 2020-06-17 2020-10-13 北京思特奇信息技术股份有限公司 Multitask visual activation method of million-magnitude data in relational database
CN112073386A (en) * 2020-08-18 2020-12-11 浙江鸿城科技有限责任公司 Window-free randomly-selectable file transmission method based on UDP (user Datagram protocol)
CN112738157A (en) * 2020-12-11 2021-04-30 苏州浪潮智能科技有限公司 File uploading method, system and equipment
CN112637341B (en) * 2020-12-22 2022-12-13 平安银行股份有限公司 File uploading method and device, electronic equipment and storage medium
CN112637341A (en) * 2020-12-22 2021-04-09 平安银行股份有限公司 File uploading method and device, electronic equipment and storage medium
CN113542264B (en) * 2021-07-13 2022-08-26 杭州安恒信息技术股份有限公司 File transmission control method, device and equipment and readable storage medium
CN113542264A (en) * 2021-07-13 2021-10-22 杭州安恒信息技术股份有限公司 File transmission control method, device, equipment and readable storage medium
CN113992938A (en) * 2021-10-22 2022-01-28 兰州乐智教育科技有限责任公司 Video uploading method and device, electronic equipment and computer readable storage medium
CN113992938B (en) * 2021-10-22 2024-04-19 兰州乐智教育科技有限责任公司 Video uploading method and device, electronic equipment and computer readable storage medium
CN114172898A (en) * 2022-02-10 2022-03-11 中国人民解放军海军工程大学 Large file remote transmission method
CN114584556A (en) * 2022-03-14 2022-06-03 中国工商银行股份有限公司 File transmission method and device
CN114630192A (en) * 2022-03-16 2022-06-14 深圳法政信息技术有限公司 Video evidence storing method
CN114598697A (en) * 2022-03-29 2022-06-07 浪潮云信息技术股份公司 Method and device for transmitting file and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN108933805A (en) A kind of document transmission method and system
CN104503745B (en) A kind of method and apparatus generated using channel packet
CN109491758A (en) Docker mirror image distribution method, system, data gateway and computer readable storage medium
CN102355426A (en) Method for transmitting off-line file and system
US9591101B2 (en) Message batching in a distributed strict queue
CN103095819A (en) Data information pushing method and data information pushing system
US9571414B2 (en) Multi-tiered processing using a distributed strict queue
US9584593B2 (en) Failure management in a distributed strict queue
CN108628881A (en) Method of data synchronization and device
US9575820B2 (en) Client control in a distributed strict queue
CN103475682A (en) File transfer method and file transfer equipment
CN104243610A (en) Distributed file transmission service method
CN106657291A (en) Data transmission method, system and electronic equipment
CN102209019A (en) Load balancing method and load balancing equipment based on message payload
CN102239680B (en) Method and device for web application hosting
US9577878B2 (en) Geographic awareness in a distributed strict queue
CN105556918B (en) A kind of resource downloading method and electronic equipment
CN109525622A (en) The generation method of fragment resource ID, resource sharing method, device and electronic equipment
CN112689020B (en) Message transmission method, message middleware, electronic equipment and storage medium
CN102629345A (en) Chain type communication cooperation method, apparatus and system thereof
KR100763526B1 (en) Device and method for management of application context
CN105245464A (en) Network acceleration method based on Android system
JP5961471B2 (en) Output comparison method in multiple information systems
US20200007608A1 (en) System and method for performing fast file transfers
CN107508787A (en) A kind of task executing method, apparatus and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20181204

RJ01 Rejection of invention patent application after publication