CN108093063A - Big file multithreading FTP method for uploading - Google Patents
Big file multithreading FTP method for uploading Download PDFInfo
- Publication number
- CN108093063A CN108093063A CN201711435751.6A CN201711435751A CN108093063A CN 108093063 A CN108093063 A CN 108093063A CN 201711435751 A CN201711435751 A CN 201711435751A CN 108093063 A CN108093063 A CN 108093063A
- Authority
- CN
- China
- Prior art keywords
- file
- upload
- files
- blocks
- uploading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to network data uploading file fields, are a kind of big file multithreading FTP method for uploading, including the first step, obtain source file；Second step monitors source file；3rd step, determines whether transmitting file；Upper transmitting file is cut into multiple blocks of files by the 4th step；5th step calculates the maximum thread of user's machine；6th step, file upload automatically；Whether the 7th step, verification file are complete；8th step, blocks of files are all transmitted, and serve end program merges blocks of files, and by file and existing system data correlation.The present invention realizes upper transmitting file automatically, without staff on duty, and supports breakpoint transmission, mistake occurs in file processes are uploaded, without user's operation, can resume file automatically after network is unobstructed, has achieved the purpose that intelligent, automation.It is uploaded than single thread uploading rate faster by multi-thread concurrent, maximumlly using the performance of client computer, upload effect is made to reach optimal, meet the requirement of client.
Description
Technical field
The present invention relates to network data uploading file fields, are a kind of big file multithreading FTP method for uploading.
Background technology
Increase popularization with mobile monitoring devices such as bayonet monitors, crime scene investigation device, recorder, instrument of taking photo by plane, video file
Management there are file inventory problem, furthermore video file be associated with existing information system structure data become it is important
Demand, it is therefore desirable to increase file upload function in existing information system.
Existing information systems most is web form system, therefore the file upload of usually web form uses http
Protocol mode is uploaded using ftp client utilities, and the video file of video monitoring equipment record is small by then hundreds of million, greatly then
Tens G even hundreds of G, but there are problems using http uploads：Because the upload of http files is disposably to load entire file
To memory, a page has once carried out file upload, this page cannot carry out other operations substantially, once network occurs
It is abnormal, it is necessary to which user reselects file is uploaded, it is low that single request uploads the rate in the case where network is bad；Use ftp
There are problems for client utility upload：Ftp client utilities can only go up transmitting file, it is impossible to close fileinfo and system information
Join, on ftp client utilities during transmitting file, network interruption occur, then need artificially to upload again manually；Most ftp visitors
Transmitting file is that single thread uploads on the ending tool of family, and efficiency of transmission is relatively low；In the event of wrong in ftp client utility transmission processes
By mistake, biography will not be mended automatically loses file.
The content of the invention
The present invention provides a kind of big file multithreading FTP method for uploading, the deficiency of the above-mentioned prior art, energy are overcome
It is low there are transmission rate effectively to solve the problems, such as that existing big file single thread is uploaded；It further solves big file to be transmitted across
Occur mistake in journey, then need to reselect the problem of file is uploaded.
The technical scheme is that realized by following measures：A kind of big file multithreading FTP method for uploading, bag
Include following steps：
The first step obtains source file；Upload service program configures a file uploaded automatically as source file, and user will treat
The file copy of upload is pressed from both sides to this document；
Second step monitors source file；Upload service program monitors this document folder by the way of polling dispatching；
3rd step, determines whether transmitting file, if so, then into the 4th step, if it is not, returning to second step；
Upper transmitting file is cut into multiple blocks of files by the 4th step；Cutting file includes procedure below：
（1）Obtain the size of file；
（2）Determine file block size；Upload service program configures each file block size according to machine performance, if need not prepare
File block size, then default settings；
（3）The file block number of cutting is calculated, formula is as follows：
The file block size of file block number=file size/configuration
The result of calculation of file block number is used into a method rounding；Enter the 5th step afterwards；
5th step calculates the maximum thread of user's machine；Obtain the free memory size of machine, according to free memory size and
The maximum thread that the file block size computing machine of configuration can be opened simultaneously, maximum thread calculation formula are as follows：
The file block size of maximum thread=machine idle memory size/configuration
The result of calculation of maximum thread is using the method rounding that truncates；
6th step, file upload automatically, including procedure below：
（1）According to the maximum thread opened simultaneously, automatic start multithreading uploads blocks of files；
（2）If（1）In all threads complete upload blocks of files, then judge blocks of files whether all upload complete, if not having
It completes, then returns（1）In continue upload blocks of files；If it completes, into the 7th step；
Whether the 7th step, verification file are complete；Whether the blocks of files for judging to have uploaded onto the server goes out active, if losing,
It returns to the 6th step and mends the blocks of files for passing and losing, completed until confirming that all blocks of files all upload；If not losing, enter
8th step；
8th step, blocks of files are all transmitted, and serve end program merges blocks of files, by file and existing system data correlation,
Terminate.
Here is the further optimization and/or improvements to foregoing invention technical solution：
It is above-mentioned to further include in the first step and judge whether this document is transmitted across when obtaining file, if not being transmitted across, into the
Two steps if being transmitted across, are ignored.
It is above-mentioned in the 6th step, after the completion of each thread uploads, serve end program updates the data upload progress, and update is completed
Afterwards, then terminate.
It is above-mentioned to further determine whether to delete source file after the completion of upload service program configuration file uploads in the 8th step,
If being configured with deletion source file, source file is deleted after the completion of upload；Do not delete source file if being configured with, return the first step after
It is continuous to obtain source file.
The present invention realizes upper transmitting file automatically, without staff on duty, and breakpoint transmission is supported, in file processes are uploaded
There is mistake, without user's operation, file can be resumed automatically after network is unobstructed, achieved the purpose that intelligent, automation.It is logical
It crosses multi-thread concurrent to upload than single thread uploading rate faster, maximumlly using the performance of client computer, imitates upload
Fruit reaches optimal, meets the requirement of client.The present invention can also verify the integrality of file, uploaded on confirmed service device
Blocks of files number and order such as find blocks of files loss situation, can play thread automatically and mend biography this document block, uploaded until all
Finish.
Description of the drawings
Attached drawing 1 is flow chart of the method for the present invention.
Specific embodiment
The present invention from following embodiments limitation, can technique according to the invention scheme and actual conditions determine specifically
Embodiment.
With reference to embodiment and attached drawing, the invention will be further described：
Embodiment 1：As shown in Figure 1, a kind of big file multithreading FTP method for uploading, comprises the following steps：
The first step obtains source file；Upload service program configures a file uploaded automatically as source file, and user will treat
The file copy of upload is pressed from both sides to this document；
Second step monitors source file；Upload service program monitors this document folder by the way of polling dispatching；
3rd step, determines whether transmitting file, if so, then into the 4th step, if it is not, returning to second step；
Upper transmitting file is cut into multiple blocks of files by the 4th step；Cutting file includes procedure below：
（1）Obtain the size of file；
（2）Determine file block size；Upload service program configures each file block size according to machine performance, if need not prepare
File block size, then default settings；
（3）The file block number of cutting is calculated, formula is as follows：
The file block size of file block number=file size/configuration
The result of calculation of file block number is used into a method rounding；Enter the 5th step afterwards；
5th step calculates the maximum thread of user's machine；Obtain the free memory size of machine, according to free memory size and
The maximum thread that the file block size computing machine of configuration can be opened simultaneously, maximum thread calculation formula are as follows：
The file block size of maximum thread=machine idle memory size/configuration
The result of calculation of maximum thread is using the method rounding that truncates；
6th step, file upload automatically, including procedure below：
（1）According to the maximum thread opened simultaneously, automatic start multithreading uploads blocks of files；
（2）If（1）In all threads complete upload blocks of files, then judge blocks of files whether all upload complete, if not having
It completes, then returns（1）In continue upload blocks of files；If it completes, into the 7th step；
Whether the 7th step, verification file are complete；Whether the blocks of files for judging to have uploaded onto the server goes out active, if losing,
It returns to the 6th step and mends the blocks of files for passing and losing, completed until confirming that all blocks of files all upload；If not losing, enter
8th step；
8th step, blocks of files are all transmitted, and serve end program merges blocks of files, by file and existing system data correlation,
Terminate.
In above-mentioned 4th step, one big video file is cut into multiple small documents according to the file block size of configuration
Block, the file block size value of configuration can make an amendment according to the performance of machine.If machine CPU is preferable, memory is relatively low, it is possible to
Using default value, if machine CPU is poor, memory is higher, then can configure the value of bigger.Such as：Conventional computer is write from memory
It is 50M to recognize value, and the file of a 10G size is cut by 50M, will cut into 10*1024/50=205 blocks of files.
In above-mentioned 5th step, maximum thread that computing machine can be opened simultaneously, such as：It is saved as in machine idle
340M, file block size are 50M, then open maximum thread as 340/50=6 thread.
In above-mentioned 6th step, multithreading is started according to the maximum thread opened simultaneously and uploads blocks of files, i.e., in file
There are no can be always maintained at all threads when being transmitted while upload to complete until file uploads.
In the 7th step, file integrality verification effect in blocks of files in transmission process it is possible that some are different
Reason condition causes blocks of files bust this to cause file lost block, in order to avoid such case, it is necessary to do file integrality verification.
If it find that there is file lost block, start thread and mend the blocks of files for passing and losing, if be all transmitted, carry out Piece file mergence.
The present invention realizes upper transmitting file automatically, without staff on duty, and breakpoint transmission is supported, in file processes are uploaded
There is mistake, without user's operation, file can be resumed automatically after network is unobstructed, achieved the purpose that intelligent, automation.It is logical
It crosses multi-thread concurrent to upload than single thread uploading rate faster, maximumlly using the performance of client computer, imitates upload
Fruit reaches optimal, meets the requirement of client.The present invention can also verify the integrality of file, uploaded on confirmed service device
Blocks of files number and order such as find blocks of files loss situation, can play thread automatically and mend biography this document block, uploaded until all
Finish.
Above-mentioned big file multithreading FTP method for uploading can be made further optimization and/or improvements according to actual needs：
As shown in Figure 1, in the first step, further include and judge whether this document is transmitted across when obtaining file, if not being transmitted across,
Then into second step, if being transmitted across, ignore.
As shown in Figure 1, in the 6th step, after the completion of each thread uploads, serve end program updates the data upload progress,
After the completion of update, then terminate.
As shown in Figure 1, in the 8th step, further determine whether to delete after the completion of upload service program configuration file uploads
Source file if being configured with deletion source file, is deleted source file after the completion of upload；Source file is not deleted if being configured with, is returned
The first step continues to obtain source file.
Here upload service program acquiescence is to delete source file, according to the needs of user, can configure and not delete source file,
Play the role of backup.
Embodiment 2：Using big file multithreading FTP method for uploading uploaded videos files, if the one of certain branch personnel
Platform computer is mounted with file uploading tools program, and configures and got well upload file, and file block size uses the configuration given tacit consent to
50M, machine idle memory 320M, upload the video file of a 12G size, and flow is as follows：
The first step, first file uploading tools scheduler program polling dispatching file, determine whether that file needs to upload, if
Hair existing file needs to upload, and carries out file cutting；
Second step, file cutting flow：Acquisition file size is 12G, and file block size is configured to 50M, then cleavable file
Block number is 12*1024/50=246 blocks of files；
3rd step, computing machine can open maximum thread flow：According to the configuration of file block size 50M and free memory
320M, maximum thread can be opened as 6 threads simultaneously by calculating；
4th step, file upload flow：After the number and the Thread Count that calculate blocks of files, start multithreading and carry out file upload,
Start 6 threads simultaneously and upload blocks of files, is i.e. the 1st thread uploads the 1st blocks of files, and the 2nd thread uploads the 2nd file
Block, and so on the 6th thread upload the 6th blocks of files, it is assumed that the 1st and 2 threads are completed simultaneously, other threads are also uploading,
Then restart two threads and upload the 7th and 8 blocks of files, i.e., remain concurrent 6 lines when file is without all having uploaded
Cheng Shangchuan, performance is preferable, update server file upload progress data after the completion of per thread uploads, for showing to user
Upload progress uses；
5th step, file integrality verification flow；After blocks of files has all uploaded, client will carry out file integrality and test
Whether card avoids the problem that losing since some abnormal conditions cause blocks of files to upload, judge on server on all files block
Completion is passed, if it find that there is file lost block, thread is again started up and mends the blocks of files for passing and losing, be until all confirming to have uploaded
Only；
6th step, server merge file, again associate file with existing system structural data after merging,
Terminate.
More than technical characteristic constitutes the embodiment of the present invention, can basis with stronger adaptability and implementation result
Actual needs increases and decreases non-essential technical characteristic, to meet the needs of different situations.
Claims (5)
1. a kind of big file multithreading FTP method for uploading, it is characterised in that comprise the following steps：
The first step obtains source file；Upload service program configures a file uploaded automatically as source file, and user will treat
The file copy of upload is pressed from both sides to this document；
Second step monitors source file；Upload service program monitors this document folder by the way of polling dispatching；
3rd step, determines whether transmitting file, if so, then into the 4th step, if it is not, returning to second step；
Upper transmitting file is cut into multiple blocks of files by the 4th step；Cutting file includes procedure below：
（1）Obtain the size of file；
（2）Determine file block size；Upload service program configures each file block size according to machine performance, if need not prepare
File block size, then default settings；
（3）The file block number of cutting is calculated, formula is as follows：
The file block size of file block number=file size/configuration
The result of calculation of file block number is used into a method rounding；Enter the 5th step afterwards；
5th step calculates the maximum thread of user's machine；Obtain the free memory size of machine, according to free memory size and
The maximum thread that the file block size computing machine of configuration can be opened simultaneously, maximum thread calculation formula are as follows：
The file block size of maximum thread=machine idle memory size/configuration
The result of calculation of maximum thread is using the method rounding that truncates；
6th step, file upload automatically, including procedure below：
（1）According to the maximum thread opened simultaneously, automatic start multithreading uploads blocks of files；
（2）If（1）In all threads complete upload blocks of files, then judge blocks of files whether all upload complete, if not having
It completes, then returns（1）In continue upload blocks of files；If it completes, into the 7th step；
Whether the 7th step, verification file are complete；Whether the blocks of files for judging to have uploaded onto the server goes out active, if losing,
It returns to the 6th step and mends the blocks of files for passing and losing, completed until confirming that all blocks of files all upload；If not losing, enter
8th step；
8th step, blocks of files are all transmitted, and serve end program merges blocks of files, by file and existing system data correlation,
Terminate.
2. big file multithreading FTP method for uploading according to claim 1, it is characterised in that in the first step, further include and obtain
Judge whether this document is transmitted across when taking file, if not being transmitted across, into second step, if being transmitted across, ignore.
3. big file multithreading FTP method for uploading according to claim 1 or 2, it is characterised in that in the 6th step, each line
After the completion of Cheng Shangchuan, serve end program updates the data upload progress, after the completion of update, then terminates.
4. big file multithreading FTP method for uploading according to claim 1 or 2, it is characterised in that in the 8th step, uploading
Service routine configuration file further determines whether to delete source file after the completion of uploading, if deletion source file is configured with, after the completion of upload
Source file is deleted；Source file is not deleted if being configured with, the first step is returned and continues to obtain source file.
5. big file multithreading FTP method for uploading according to claim 3, it is characterised in that in the 8th step, taken uploading
Business program configuration file further determines whether to delete source file after the completion of uploading, right after the completion of upload if being configured with deletion source file
Source file is deleted；Source file is not deleted if being configured with, the first step is returned and continues to obtain source file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711435751.6A CN108093063A (en) | 2017-12-26 | 2017-12-26 | Big file multithreading FTP method for uploading |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711435751.6A CN108093063A (en) | 2017-12-26 | 2017-12-26 | Big file multithreading FTP method for uploading |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108093063A true CN108093063A (en) | 2018-05-29 |
Family
ID=62179479
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711435751.6A Pending CN108093063A (en) | 2017-12-26 | 2017-12-26 | Big file multithreading FTP method for uploading |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108093063A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324897A (en) * | 2018-08-24 | 2019-02-12 | 平安科技（深圳）有限公司 | Data uploading method and system, terminal and computer readable storage medium |
| CN111767331A (en) * | 2020-06-17 | 2020-10-13 | 北京思特奇信息技术股份有限公司 | Multitask visual activation method of million-magnitude data in relational database |
| CN112511633A (en) * | 2020-12-03 | 2021-03-16 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN113824760A (en) * | 2021-08-03 | 2021-12-21 | 中债金科信息技术有限公司 | File concurrent block uploading method and device, electronic equipment and storage medium |
| CN114979250A (en) * | 2022-04-18 | 2022-08-30 | 北京嗨学网教育科技股份有限公司 | Message pushing method, device and equipment |
| CN114979250B (en) * | 2022-04-18 | 2024-06-04 | 北京嗨学网教育科技股份有限公司 | Message pushing method, device and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103442037A (en) * | 2013-08-09 | 2013-12-11 | 华南理工大学 | Method for achieving multithreading breakpoint upload of oversized file based on FTP |
| CN104038498A (en) * | 2014-06-13 | 2014-09-10 | 大同煤矿集团有限责任公司 | File synchronization updating method and device |
| CN105338090A (en) * | 2015-11-12 | 2016-02-17 | 浪潮软件集团有限公司 | WebSocket-based breakpoint continuous transmission method and device |
| US20170171295A1 (en) * | 2015-12-09 | 2017-06-15 | Synology Incorporated | File upload and download methods and associated server |
| CN107450997A (en) * | 2017-07-27 | 2017-12-08 | 东软集团股份有限公司 | The method and device of picture dynamic play |
-
2017
- 2017-12-26 CN CN201711435751.6A patent/CN108093063A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103442037A (en) * | 2013-08-09 | 2013-12-11 | 华南理工大学 | Method for achieving multithreading breakpoint upload of oversized file based on FTP |
| CN104038498A (en) * | 2014-06-13 | 2014-09-10 | 大同煤矿集团有限责任公司 | File synchronization updating method and device |
| CN105338090A (en) * | 2015-11-12 | 2016-02-17 | 浪潮软件集团有限公司 | WebSocket-based breakpoint continuous transmission method and device |
| US20170171295A1 (en) * | 2015-12-09 | 2017-06-15 | Synology Incorporated | File upload and download methods and associated server |
| CN107450997A (en) * | 2017-07-27 | 2017-12-08 | 东软集团股份有限公司 | The method and device of picture dynamic play |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324897A (en) * | 2018-08-24 | 2019-02-12 | 平安科技（深圳）有限公司 | Data uploading method and system, terminal and computer readable storage medium |
| CN111767331A (en) * | 2020-06-17 | 2020-10-13 | 北京思特奇信息技术股份有限公司 | Multitask visual activation method of million-magnitude data in relational database |
| CN112511633A (en) * | 2020-12-03 | 2021-03-16 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN112511633B (en) * | 2020-12-03 | 2022-11-29 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for block transmission of massive small files |
| CN113824760A (en) * | 2021-08-03 | 2021-12-21 | 中债金科信息技术有限公司 | File concurrent block uploading method and device, electronic equipment and storage medium |
| CN113824760B (en) * | 2021-08-03 | 2024-01-30 | 中债金科信息技术有限公司 | File concurrency block uploading method and device, electronic equipment and storage medium |
| CN114979250A (en) * | 2022-04-18 | 2022-08-30 | 北京嗨学网教育科技股份有限公司 | Message pushing method, device and equipment |
| CN114979250B (en) * | 2022-04-18 | 2024-06-04 | 北京嗨学网教育科技股份有限公司 | Message pushing method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108093063A (en) | Big file multithreading FTP method for uploading | |
| CN111031094A (en) | Data transmission method, device, equipment and storage medium in IoT system | |
| CN103875218B (en) | Content reproduction information estimating unit, method | |
| CN103699063B (en) | The harvester of off-line data and method in a kind of Manufacturing Executive System MES | |
| CN111970195B (en) | Data transmission method and streaming data transmission system | |
| CN108347463A (en) | A kind of terminal document method for uploading | |
| WO2021147793A1 (en) | Data processing method, apparatus and system, electronic device and computer storage medium | |
| CN105388751A (en) | Method and system used for preventing clock jump in motor train unit | |
| CN103188305A (en) | Multithreading breakpoint resume method and system | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN105721526A (en) | Terminal and server file synchronization methods and device | |
| CN105472421A (en) | Video program collecting, editing and making method and system based on public cloud | |
| CN105718569A (en) | Method, device and system for uploading compound document | |
| CN102300081B (en) | Audio and video monitoring equipment, system and method thereof | |
| CN105607606A (en) | Data acquisition device and data acquisition method based on double-mainboard framework | |
| CN103929455A (en) | Asynchronous file transmission system and method comprising network storage equipment | |
| WO2011134300A1 (en) | Device, method and system for processing machine to machine service | |
| CN117278653A (en) | Data record transmission system and method for distributed site satellite | |
| CN102075385A (en) | Disk array distributed test method | |
| CN105516351B (en) | A method of realizing concentrator remote upgrade multi-threaded parallel | |
| CN103795810A (en) | Data distribution system and method and central server of data distribution system | |
| CN105429823A (en) | Method and device for detecting multicast traffic in distributed communication equipment | |
| CN113691520A (en) | Method, device, storage medium and electronic device for acquiring streaming media information | |
| CN209803613U (en) | Plastic mechanical equipment monitoring device and system | |
| CN113507497A (en) | Multi-node asynchronous issuing and delay integration method for multi-type data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180529 |