CN110505298A - Optimization system and its method based on transmitting file on linux system - Google Patents
Optimization system and its method based on transmitting file on linux system Download PDFInfo
- Publication number
- CN110505298A CN110505298A CN201910785356.3A CN201910785356A CN110505298A CN 110505298 A CN110505298 A CN 110505298A CN 201910785356 A CN201910785356 A CN 201910785356A CN 110505298 A CN110505298 A CN 110505298A
- Authority
- CN
- China
- Prior art keywords
- file
- module
- cutting
- client
- transmitting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Abstract
The invention discloses a kind of optimization systems based on transmitting file on linux system, including the fractionation module and sending module positioned at client and positioned at the integration module of server end, in which: the asynchronous fractionation of file progress split after module is used to parse client；The sending module is for the file after splitting by client asynchronous transmission to server end；The module of integrating is for assembling the file that server end receives.The invention also discloses a kind of optimization methods based on transmitting file on linux system；The present invention achievees the purpose that shorten uplink time by optimizing the above fractionation mode of transmitting file and the mode of asynchronous upload file.
Description
Technical field
The present invention relates to software technology field, especially a kind of optimization system based on transmitting file on linux system and its
Method.
Background technique
(SuSE) Linux OS passes through the development of more than ten years, a very outstanding operating system has been grown into now, not
There is increasingly wider application in same field.Existing upper transmitting file prioritization scheme is mostly to retransmit after handling file failure, processing
It is the problem of upper transmitting file is not disrupted；Biography prioritization scheme be serially to be stored after handling file fragmentation, by file data
It is divided at least one data block in order；After determining first data block that file data marks off, be this document block into
A series of Hash encryptions of row etc. generate blocks of files ID and directly store.When the reason of storage file block is that user downloads file
System assembles blocks of files into original document again.It is most of to guarantee entire text to guarantee the success of web server deploying projects
The case where part full storage, while cutting memory block in order to blocks of files when uploading and inherently consuming the more time,
And what is stored is the data block after fragment, and readability is deteriorated, and also can not more easily check file content on the server.
Current most of corporate servers use linux system, and the different size of file of analysis upload compares when uploading
Between it can be found that upload the time that bigger file waits longer.
Summary of the invention
To solve problems of the prior art, the object of the present invention is to provide one kind uploads text based on linux system
The optimization system and its method of part, the present invention are reached by optimizing the above fractionation mode of transmitting file and the mode of asynchronous upload file
To the purpose for shortening uplink time.
To achieve the above object, the technical solution adopted by the present invention is that: a kind of optimization based on transmitting file on linux system
System, including the fractionation module and sending module positioned at client and positioned at the integration module of server end, in which:
The asynchronous fractionation of file progress split after module is used to parse client；
The sending module is for the file after splitting by client asynchronous transmission to server end；
The module of integrating is for assembling the file that server end receives.
As a preferred embodiment, the fractionation module is directly opened multiple if what is uploaded is single file
Sub thread concurrently cuts file, if what is uploaded is file, the fractionation module is to each of file
File concurrently cuts file using multiple sub threads.
As another preferred embodiment, the module that splits is cut according to the original file content size formulation of upload
The starting point of data number of blocks and each cutting after cutting, data number of blocks is the number for opening sub thread, for each
A sub thread carries out the operation of cutting file all in accordance with respective cutting starting point.
As another preferred embodiment, the fractionation module split to file and be specifically included:
The byte array for creating a specified size, opens multiple sub threads, and sub thread quantity is file size divided by list
A byte size of data, each sub thread read the file after cutting in byte array, when all sub threads have executed
Enter the sending module after finishing.
As another preferred embodiment, the filename of the file after cutting include upload file filename and
Number is numbered from 1 to n, and wherein n is the file number after cutting.
As another preferred embodiment, the sending module passes through the file number and file after judgement cutting
The sub thread that size opens different numbers concurrently sends file, and in transmission process by socket program realize client and
The communication of server end.
As another preferred embodiment, it is described integrate module the file that server end receives assemble it is specific
Include:
The file that client is sent is received, and is grouped according to each filename, if the filename of prefix is the same
It then indicates to belong to same file, in the case where multiple files, each sub thread is the process of the different files of an integration, son
Thread defines a byte array first, and file content is successively written to the byte array of definition according to the number in filename,
Byte array is finally written in big file to the assembling for completing file.
The present invention also provides a kind of optimization methods based on transmitting file on linux system, comprising the following steps:
Step 1: user selects the file to be uploaded, and file is parsed；
Step 2: judging file size into splitting step and being cut to it, opened according to the quantity of documents of cutting
Multiple sub threads of corresponding number create multiple FileOutputStream () objects, it is therefore an objective to which creating multiple filenames includes
The filename of the file of upload and the empty file of number, for successively the byte data after cutting to be written wherein；
Step 3: each sub thread initially sets up socket object instance, getOutputStream () method, mesh are called
Be to establish data buffer zone, for store to server end send data, transmit data to server end；Creation
FilelnputStream object instance, file to be sent is written in output stream data buffer zone, after being finally sent
Close each connection；
Step 4: to create communication connection, during which the example of creation ServerSocket object calls accept () method
It experienced TCP three-way handshake, call getInputStream () method to receive the file of client transmission, call
FileOutputStream file output stream is used to write out data, successively traverses file from small to large according to reference number of a document, creates
One byte array is used to store the file of client transmission, and byte array is successively output to FileOutputStream pairs
As the FileOutputStream object of generation is the file completed after integration.
As a preferred embodiment, in step 1, if what is uploaded is that single file or multiple files are straight
It taps into splitting step；If detect client upload is file, order Runtime.getRuntime is first transferred
() .exec creates a file of the same name in server.
The beneficial effects of the present invention are: file this part that the present invention uploads needs is optimized, effectively shorten
The time of entire upper transmitting file, the speed of the transmitting file on linux system can be accelerated, save developer deploying projects and spend
Time when making user that need not wait more when uploading big file, under the premise of guaranteeing that file is complete and consistent with before upload, uses
The time of transmitting file is substantially reduced on family, enhances user experience.
Detailed description of the invention
Fig. 1 is the functional block diagram of the embodiment of the present invention.
Specific embodiment
The embodiment of the present invention is described in detail with reference to the accompanying drawing.
Embodiment
As shown in Figure 1, a kind of optimization system based on transmitting file on linux system, the fractionation mould including being located at client
Block and sending module and integration module positioned at server end, in which:
1, it splits module: by the asynchronous fractionation of file after parsing, solving the excessive caused time-consuming of big file content, and
It is serial to split caused program obstruction；
2, sending module: sending time is too long when the file asynchronous transmission after fractionation is solved the problems, such as that file is excessive；
3, integrate module: the file that server end assembling receives ensure that the integrality of file；
Split module:
It is not identical for single file and file deconsolidation process: if what is uploaded is that single file directly opens multiple lines
Journey (mainly carrying out operation concurrently with thread herein, to save and split the time) is split file；If what is uploaded is text
Part folder then needs to be split each file in file, is equally handled using multi-thread concurrent.Fractionation scheme be according to
Original file content size formulates the data number of blocks after cutting and the starting point of each cutting, data number of blocks are exactly
The number for opening sub thread carries out the operation of cutting file for each sub thread all in accordance with respective cutting starting point.
Specific cutting scheme is the byte array for creating a specified size, opens multithreading, and number of threads is that file is big
It is small divided by single byte array size, each thread concrete operations is to read big file in byte array, when all sons
Thread is finished into transmission file module.At this time the last one small documents may memory size as defined in deficiency, from big
When file reads the last one byte array, may also and not read full byte array, big file attend school take it is complete, so can only will
The last regulation memory size byte array that do not meet is written in the last one file, should be noted the text of the small documents of each fractionation
Filename+number of the entitled big file of part is numbered from 1 to n, and n is the file number after cutting.
Each sub thread be carried out after then cutting operation just calculate completion.Different from the place of existing cutting file
It is multi-thread concurrent processing cutting file, while simplifies the calculating process for data block ID, currently for the number after cutting
According to block directly according to upload filename+number form name.
Sending module:
By processing mode of the file asynchronous transmission for single file such as file after fractionation, pass through judgement text
The thread that the number and file size of part open different numbers concurrently sends file；About the mode for sending file: tradition hair
Sending file, there are two types of modes: a, the window that the upper transmitting file of selection can be directly transferred in Linux command row interface input " rz " order
Mouthful；B, installation xftp plug-in unit is uploaded.Above two mode all can only single file send or multifile serially send.But
It is to realize client kimonos using socket programming in the present invention due to needing the file to cutting to carry out asynchronous transmission
The communication at business device end, is then turned on multithreading asynchronous transmission file, and socket connection is disconnected after being sent.The multithreading of unlatching
In, the transmission of the corresponding sub thread of a file.
Integrate module:
The file received is assembled about server end in module is integrated, is different from the first two module, the first two module herein
In the same script and it is located at client, and integrates module and be located at server end, and file and folder content are answered after integration
It is consistent with when sending.The file that client is sent is received, and is grouped according to each filename, if prefix is big
Filename is the same, and expression belongs to same file, and in the case where multiple files, each sub thread is an integration not identical text
The process of part.The groundwork of sub thread is to define one big byte array, successively will be in file according to the number in filename
Hold write-in byte array, finally byte array is written in big file.
The present embodiment also provides a kind of optimization method based on transmitting file on linux system, comprising the following steps:
Step 1: user selects the file to be uploaded, and the file of selection is parsed.If what is uploaded is single text
Part or multiple files are directly entered splitting step；If detect client upload is file, order is first transferred
Runtime.getRuntime () .exec (" mkdir folder name ") creates a file of the same name in server end.
Step 2: entering splitting step, judge file size, if current file size is 10M, creates prescribed level
Byte array is 1024KB, then Thread Count is 10M/1024KB=10, and first file starting point is 0, and second is 1024,
Third is 1024*2 ....Open multithreading, for first sub thread cut coverage be 0-1024, direct time
The 0-1024 byte data of big file is gone through, creates FileOutputStream () object, it is therefore an objective to which building a name is
The empty file of " filename+1 ", then the byte data of the big file of 0-1024 is written.Second thread is repeated no more, only
Be cut coverage be 1024-2048, and generate name be " filename+2 ".It is finished when 10 sub threads are carried out, generates 10
A data block file, and filename is respectively filename 1, filename 2, filename 3, filename 4, filename 5, filename 6, text
Part name 7, filename 8, filename 9, filename 10.
Step 3: 10 threads, the corresponding sub thread of a data block file are opened.Each sub thread is initially set up
Socket object instance calls getOutputStream () method, it is therefore an objective to data buffer zone is established, for storing to service
The data that device end is sent, transmit data to server end.Create FileInputStream object instance, text to be sent
Part is written in output stream data buffer zone.Each connection is closed after being finally sent.
Step 4: to create communication connection, during which the example of creation ServerSocket object calls accept () method
It experienced TCP three-way handshake, call getInputStream () method to receive the file of client transmission.It calls
FileOutputStream file output stream is used to write out data.File is successively traversed from small to large according to reference number of a document, is created
One byte array is used to store the file of client transmission, and byte is successively output to FileOutputStream object.
The FileOutputStream object of generation is the file completed after integration.
A specific embodiment of the invention above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.
Claims (9)
1. a kind of optimization system based on transmitting file on linux system, which is characterized in that the fractionation module including being located at client
With sending module and positioned at the integration module of server end, in which:
The asynchronous fractionation of file progress split after module is used to parse client；
The sending module is for the file after splitting by client asynchronous transmission to server end；
The module of integrating is for assembling the file that server end receives.
2. the optimization system according to claim 1 based on transmitting file on linux system, which is characterized in that if uploaded
Be single file, then the fractionation module is directly opened multiple sub threads and is concurrently cut to file, if upload be text
Part folder, then the fractionation module concurrently cuts each of file file using multiple sub threads to file.
3. the optimization system according to claim 2 based on transmitting file on linux system, which is characterized in that the fractionation
Module formulates the data number of blocks after cutting and the starting point of each cutting according to the original file content size of upload,
Data number of blocks is that the number of unlatching sub thread is cut for each sub thread all in accordance with respective cutting starting point
The operation of file.
4. the optimization system according to claim 3 based on transmitting file on linux system, which is characterized in that the fractionation
Module split to file and be specifically included:
The byte array for creating a specified size, opens multiple sub threads, and sub thread quantity is file size divided by single
Byte size of data, each sub thread read the file after cutting in byte array, when all sub threads are finished
Enter the sending module afterwards.
5. the optimization system according to claim 4 based on transmitting file on linux system, which is characterized in that after cutting
The filename of file includes the filename and number of the file uploaded, is numbered from 1 to n, and wherein n is the file number after cutting.
6. the optimization system according to claim 5 based on transmitting file on linux system, which is characterized in that the transmission
Module concurrently sends file by the sub thread that file number after judgement cutting and file size open different numbers, and
The communication for realizing client and server end is programmed in transmission process by socket.
7. the optimization system according to claim 5 or 6 based on transmitting file on linux system, which is characterized in that described whole
Molding block carries out assembling to the file that server end receives and specifically includes:
The file that client is sent is received, and is grouped according to each filename, the table if the filename of prefix is the same
Show and belong to same file, in the case where multiple files, each sub thread is the process of the different files of an integration, sub thread
A byte array is defined first, according to the number in filename successively by the byte array of file content write-in definition, finally
Byte array is written in big file to the assembling for completing file.
8. a kind of optimization method based on transmitting file on linux system, which comprises the following steps:
Step 1: user selects the file to be uploaded, and file is parsed；
Step 2: judging file size into splitting step and being cut to it, is opened and corresponded to according to the quantity of documents of cutting
Multiple sub threads of quantity create multiple FileOutputStream () objects, it is therefore an objective to which creating multiple filenames includes uploading
File filename and number empty file, for successively by the byte data write-in after cutting wherein；
Step 3: each sub thread initially sets up socket object instance, getOutputStream () method is called, it is therefore an objective to
Data buffer zone is established, for storing the data sent to server end, transmits data to server end；Creation
FileInputStream object instance, file to be sent is written in output stream data buffer zone, after being finally sent
Close each connection；
Step 4: the example of creation ServerSocket object is called accept () method to create communication connection, is during which undergone
TCP three-way handshake is called getInputStream () method to receive the file of client transmission, is called
FileOutputStream file output stream is used to write out data, successively traverses file from small to large according to reference number of a document, creates
One byte array is used to store the file of client transmission, and byte array is successively output to FileOutputStream pairs
As the FileOutputStream object of generation is the file completed after integration.
9. the optimization method according to claim 8 based on transmitting file on linux system, which is characterized in that in step 1
In, if what is uploaded is that single file or multiple files are directly entered splitting step；If detect client upload is
File then first transfers order Runtime.getRuntime () .exec in server and creates a file of the same name.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910785356.3A CN110505298A (en) | 2019-08-23 | 2019-08-23 | Optimization system and its method based on transmitting file on linux system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910785356.3A CN110505298A (en) | 2019-08-23 | 2019-08-23 | Optimization system and its method based on transmitting file on linux system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110505298A true CN110505298A (en) | 2019-11-26 |
Family
ID=68589343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910785356.3A Pending CN110505298A (en) | 2019-08-23 | 2019-08-23 | Optimization system and its method based on transmitting file on linux system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110505298A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111314449A (en) * | 2020-02-06 | 2020-06-19 | 北京东方金信科技有限公司 | File breakpoint continuous transmission method and system |
| CN111813413A (en) * | 2020-06-28 | 2020-10-23 | 四川长虹电器股份有限公司 | Method for automatically generating YAML file |
| CN114301900A (en) * | 2021-12-27 | 2022-04-08 | 中国电信股份有限公司 | File transmission method and device and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102063334A (en) * | 2011-01-17 | 2011-05-18 | 浪潮(北京)电子信息产业有限公司 | Method and device for quickly copying file |
| CN103841181A (en) * | 2014-01-24 | 2014-06-04 | 东蓝数码股份有限公司 | Local area network multi-threading file transmission method |
| CN103997514A (en) * | 2014-04-23 | 2014-08-20 | 汉柏科技有限公司 | File parallel transmission method and system |
| CN104394229A (en) * | 2014-12-09 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Large file uploading method based on concurrent transmission mode |
| CN106060160A (en) * | 2016-07-07 | 2016-10-26 | 腾讯科技（深圳）有限公司 | File uploading method and device |
| CN108933805A (en) * | 2017-05-26 | 2018-12-04 | 武汉斗鱼网络科技有限公司 | A kind of document transmission method and system |
-
2019
- 2019-08-23 CN CN201910785356.3A patent/CN110505298A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102063334A (en) * | 2011-01-17 | 2011-05-18 | 浪潮(北京)电子信息产业有限公司 | Method and device for quickly copying file |
| CN103841181A (en) * | 2014-01-24 | 2014-06-04 | 东蓝数码股份有限公司 | Local area network multi-threading file transmission method |
| CN103997514A (en) * | 2014-04-23 | 2014-08-20 | 汉柏科技有限公司 | File parallel transmission method and system |
| CN104394229A (en) * | 2014-12-09 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Large file uploading method based on concurrent transmission mode |
| CN106060160A (en) * | 2016-07-07 | 2016-10-26 | 腾讯科技（深圳）有限公司 | File uploading method and device |
| CN108933805A (en) * | 2017-05-26 | 2018-12-04 | 武汉斗鱼网络科技有限公司 | A kind of document transmission method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111314449A (en) * | 2020-02-06 | 2020-06-19 | 北京东方金信科技有限公司 | File breakpoint continuous transmission method and system |
| CN111813413A (en) * | 2020-06-28 | 2020-10-23 | 四川长虹电器股份有限公司 | Method for automatically generating YAML file |
| CN114301900A (en) * | 2021-12-27 | 2022-04-08 | 中国电信股份有限公司 | File transmission method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110505298A (en) | Optimization system and its method based on transmitting file on linux system | |
| CN108933805A (en) | A kind of document transmission method and system | |
| EP2887583B1 (en) | Service processing method and device | |
| CN103259797B (en) | data file transmission method and platform | |
| CN105786549B (en) | Smart machine and its upgrade method | |
| CN103227836B (en) | Offline download method and off-line download server | |
| CN103714137B (en) | The delet method and system of a kind of data file | |
| CN107689976B (en) | File transmission method and device | |
| CN110347747A (en) | Database data synchronic method, system, computer equipment and storage medium | |
| CN112328565A (en) | Resource sharing method and device based on block chain | |
| CN110336879A (en) | A kind of method and apparatus of the data processing based on edge calculations | |
| CN104079623B (en) | Multistage cloud storage synchronisation control means and system | |
| CN107222554A (en) | A kind of document transmission method and system | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN110705891A (en) | Data processing method based on high-allocable changeability | |
| CN101335763B (en) | Data educing and transmitting method in intelligent network system | |
| CN104113594A (en) | Method and system for file uploading based on JS uploading assembly | |
| CN110022287A (en) | A kind of asynchronous distributed document transmission method, terminal device and storage medium | |
| CN110362548A (en) | Log extracting method and relevant device | |
| CN109684274A (en) | Archive method, equipment and computer readable storage medium | |
| CN107291610A (en) | Testing case management and device | |
| CN112836162B (en) | Content delivery method and system | |
| CN108287920A (en) | A method of handling office files under BS systems | |
| CN109766377A (en) | Cross-platform data capture method, device, computer equipment and storage medium | |
| CN111159752B (en) | Method and device for sharing information between mechanisms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191126 |
|
| RJ01 | Rejection of invention patent application after publication |