CN104426942A - File uploading method and system - Google Patents
File uploading method and system Download PDFInfo
- Publication number
- CN104426942A CN104426942A CN201310377622.1A CN201310377622A CN104426942A CN 104426942 A CN104426942 A CN 104426942A CN 201310377622 A CN201310377622 A CN 201310377622A CN 104426942 A CN104426942 A CN 104426942A
- Authority
- CN
- China
- Prior art keywords
- file
- server
- block
- client
- cutting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
Abstract
The invention provides a file uploading method, which comprises steps: a file needed to be uploaded to a server is acquired at a client, and the transmission speed between the client and the server is acquired; the file is divided into multiple blocks according to the transmission speed; the multiple blocks are transmitted to the server through parallel transmission; and after the server receives the multiple blocks transmitted by the client, the multiple received blocks are compressed in a parallel calculation mode and the multiple compressed blocks are stored in a distributive file system of the server. The invention also provides a file uploading system. Thus, the uploading efficiency of the file is improved.
Description
Technical field
The present invention relates to a kind of file uploading method and system, especially relate to a kind of method and system by webpage upload file.
Background technology
Along with the development of science and technology, high in the clouds stores with the application of distributed file system increasingly mature, and one of main trend becoming network.In the client, user can by webpage upload file in server.During usual file is all uploaded onto the server once, now, if the file size uploaded is larger, then the transmission of file can just be completed through long transmission.In this long transmitting procedure, the network between client and server easily interrupts, then file needs to re-start transmission, thus causes the upper transfer efficiency of file low.
Summary of the invention
In view of above content, be necessary to provide a kind of file uploading method and system, it can utilize file block, parallel transmission, parallel computation and file compression techniques to promote the upper transfer efficiency of file.
A kind of file uploading method, the method comprising the steps of: (a) obtains the file needing to upload onto the server in the client, and obtains the transmission speed between client and server; B file cutting is multiple block according to described transmission speed by (); (c) by parallel transmission by described multiple encrypted communication in server; And (d) is after server receives multiple blocks of client transmissions, compresses multiple block of this reception by the mode of parallel computation and multiple blocks of this compression are stored in the distributed file system of server.
A kind of file uploading system, run in client, this system comprises: acquisition module, for obtaining the file needing to upload onto the server, and obtains the transmission speed between client and server; Cutting module, for being multiple block according to described transmission speed by file cutting; And transport module, for by parallel transmission by described multiple encrypted communication in server, described server, after receiving multiple blocks of client transmissions, compresses multiple block of this reception by the mode of parallel computation and is stored in the distributed file system of server by multiple blocks of this compression.
A kind of file uploading system, run in server, this system comprises: compression memory module, for receiving multiple blocks of client transmissions, described multiple block is according to the transmission speed between server by client, to the file uploaded onto the server be needed to carry out cutting, and transfer to server by parallel transmission; Described compression memory module, also for being compressed multiple block of this reception by the mode of parallel computation and being stored in the distributed file system of server by multiple blocks of this compression.
Compared to prior art, file uploading method provided by the invention and system utilize file block, parallel transmission, parallel computation and file compression techniques, can be multiple block by file cutting in client, and by the plurality of block parallel transmission in server, and by parallel computation described multiple block compressed in the server and be stored in the distributed file system of server, the upper transfer efficiency of file is promoted with this.
Accompanying drawing explanation
Fig. 1 is the network architecture diagram of server of the present invention and client.
Fig. 2 is the functional block diagram of file uploading system preferred embodiment of the present invention.
Fig. 3 is the operation process chart of file uploading method preferred embodiment of the present invention.
Main element symbol description
| Client | 100 |
| File uploading system | 10 |
| Acquisition module | 11 |
| Cutting module | 12 |
| Transport module | 13 |
| Compression memory module | 14 |
| Memory cell | 20 |
| Network | 200 |
| Server | 300 |
| Distributed file system | 30 |
| Database | 40 |
Following embodiment will further illustrate the present invention in conjunction with above-mentioned accompanying drawing.
Embodiment
Consulting shown in Fig. 1, is the network architecture diagram of server of the present invention and client.In this preferred embodiment, described client 100 is connected with server 300 by network 200.Described network 200 can be intranet (Intranet), also can be the communication network of the Internet (Internet) or other type.Described client 100 can be the terminal such as notebook computer or computer.Described server 300 is connected with database 40 by DataBase combining, this DataBase combining can be JDBC(Java Database Connectivity, Java DataBase combining) or ODBC(Open Database Connectivity, open CNC) etc.Described client 100 comprises memory cell 20, and this memory cell 20 is for storing the file to server 300 to be uploaded.Described server 300 comprises distributed file system 30, and this distributed file system 30 transfers to file in server 300 for storing client 100.
File uploading system 10 is installed in client 100 and server 300.In client 100, this file uploading system 10 needs for obtaining the upload onto the server file of 300 and the transmission speed between client 100 and server 300, and be multiple block according to this transmission speed by file cutting, and by parallel transmission by the plurality of encrypted communication in server 300.When server 300 receives multiple block of client 100 transmission, multiple blocks after this compression for being compressed multiple blocks of this reception by the mode of parallel computation, and are stored in described distributed file system 30 by the mode of parallel computation by described file uploading system 10.
Consulting shown in Fig. 2, is the functional block diagram of file uploading system preferred embodiment of the present invention.This file uploading system 10 comprises the computerization program command be installed in client 100 and server 300.Described file uploading system 10 comprises acquisition module 11, cutting module 12, transport module 13 and compression memory module 14.Wherein, described acquisition module 11, cutting module 12 and transport module 13 are applied in client 100, and described compression memory module 14 is applied in server 300.
Acquisition module 11 for obtain user by needing of selecting of the webpage of client 100 upload onto the server 300 file, and obtain the transmission speed between client 100 and server 300.In this preferred embodiment, the file be stored in memory cell 20 is selected by the user to carry out upload operation.
Cutting module 12 is for being multiple block according to described transmission speed by file cutting.In this preferred embodiment, cutting module 12 receives the block worst case transmit time of user's definition, and utilize the size of each block after described transmission speed and the cutting of block worst case transmit time determination file, and utilize the size cutting file of each block after this cutting.
It should be noted that, when network 200 continuous data transfer is to server 300, if the transmission time is more than a Preset Time, network 200 easily interrupts, then cause data needs again to be transmitted, thus affect efficiency of transmission.Described block worst case transmit time is the numerical value being less than or equal to described Preset Time, and this Preset Time is determined based on experience value by user.Such as Preset Time is 5 minutes, then described block worst case transmit time is less than or equal to 5 minutes.
Specifically, described transmission speed be multiplied with the block worst case transmit time numerical value that obtains of cutting module 12 is defined as the size of each block after file cutting.Such as, block worst case transmit time is 4 minutes, and described transmission speed is 100Kb/s (kilobits per second), and the size of each block after the file cutting obtained after calculating is 2.93MB(Mbytes).If file is 10MB, then this file cutting is the block of three 2.93MB and the block of a 1.21MB by cutting module 12.
Described cutting module 12 is also for generating the block index of described multiple block, and this block index may be used for described multiple block to be merged into a complete file.
Transport module 13 for by parallel transmission by described multiple encrypted communication in server 300.
Compression memory module 14 for receive at server 300 client 100 transmit multiple blocks after, compress multiple block of this reception by the mode of parallel computation and multiple blocks of this compression be stored in distributed file system 30.
In this preferred embodiment, compression memory module 14 is when using parallel computation, first start a thread to compress a block and the block of this compression is stored in distributed file system 30, afterwards, under the prerequisite of server 300 resource permission, parallel thread is increased to compress block and to be stored in distributed file system 30 by the block of compression.Described compression memory module 14 regulates Thread Count automatically according to the load state of the processor of server 300, until the utilization rate of processor reaches 100%, wherein, each thread all can process compression and the storage operation of different blocks, thus can increase the memory rate of file.
Due to when compressing multiple block, block compressed action can take a large amount of system resources of server 300.In other embodiments of the invention, compression memory module 14 is only selected jumbo file and is carried out squeeze operation, and does not compress the file of low capacity, is directly stored in distributed file system 30, exceeds load to avoid server 300.Particularly, compress memory module 14 and judge whether the capacity of file is greater than a preset capacity, to determine whether the multiple blocks compressing this file.Described preset capacity is arranged based on experience value by user, as being 1GB(gigabyte) or the numerical value such as 2GB.Such as, compression memory module 14, when file is greater than a preset capacity, carries out the squeeze operation of multiple blocks of this file.Compression memory module 14, when file is not more than a preset capacity, is not carried out the squeeze operation of multiple blocks of this file, is directly stored in distributed file system 30 by multiple blocks of this file by the mode of parallel computation.
Described compression memory module 14 is also sent to the relevant information of the file of server 300 for receiving client 100, and the relevant information of this file is stored in database 40.The relevant information of described file is the information such as the block index of file, for when the multiple block of downloaded stored in distributed file system 30, the plurality of block is combined into complete file.
The present invention utilizes file block and parallel transmission to promote the efficiency of transmission of file, and utilize parallel computation and compressing file to increase the memory rate that file is stored to distributed file system 30, thus file can be uploaded onto the server in the distributed file system 30 of 300 fast, improve the upper transfer efficiency of file.
Consulting shown in Fig. 3, is the flow chart of file uploading method preferred embodiment of the present invention.
Step S10, acquisition module 11 obtain user by needing of selecting of the webpage of client 100 upload onto the server 300 file, and obtain the transmission speed between client 100 and server 300.
Step S20, file cutting is multiple block according to described transmission speed by cutting module 12.In this preferred embodiment, cutting module 12 receives the block worst case transmit time of user's definition, and utilize the size of each block after described transmission speed and the cutting of block worst case transmit time determination file, and utilize the size cutting file of each block after this cutting.Such as, block worst case transmit time is 4 minutes, and described transmission speed is 100Kb/s (kilobits per second), and the file obtained after calculating wants the block size of cutting to be 2.93MB(Mbytes).If file is 10MB, then this file cutting is the block of three 2.93MB and the block of a 1.21MB by cutting module 12.
In step S20, cutting module 12 also generates the block index of described multiple block, and this block index may be used for described multiple block to be merged into a complete file.
Step S30, transport module 13 by parallel transmission by described multiple encrypted communication in server 300.
Step S40, compression memory module 14 is compressed multiple block of this reception by the mode of parallel computation and is stored in distributed file system 30 by multiple blocks of this compression receive multiple blocks of client 100 transmission at server 300 after.
In other embodiments of the invention, compress memory module 14 and judge whether the capacity of file is greater than a preset capacity, to determine whether the multiple blocks compressing this file.Described preset capacity is arranged based on experience value by user, as being 1GB(gigabyte) or the numerical value such as 2GB.Such as, compression memory module 14, when file is greater than a preset capacity, carries out the squeeze operation of multiple blocks of this file.Compression memory module 14, when file is not more than a preset capacity, is not carried out the squeeze operation of multiple blocks of this file, is directly stored in distributed file system 30 by multiple blocks of this file by the mode of parallel computation.
After the step s 40, described compression memory module 14 receives the relevant information that client 100 is sent to the file of server 300, and the relevant information of this file is stored in database 40.The relevant information of described file is the information such as the block index of file, for when the multiple block of downloaded stored in distributed file system 30, the plurality of block is combined into complete file.
It should be noted last that, above embodiment is only in order to illustrate technical scheme of the present invention and unrestricted, although with reference to above preferred embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that, can modify to technical scheme of the present invention or equivalent replacement, and not depart from the spirit and scope of technical solution of the present invention.
Claims (11)
1. a file uploading method, is characterized in that, the method comprises:
Obtaining step: obtain the file needing to upload onto the server in the client, and obtain the transmission speed between client and server;
Dicing step: be multiple block by file cutting according to described transmission speed;
Transmitting step: by parallel transmission by described multiple encrypted communication in server; And
Compression storing step: receive multiple blocks of client transmissions at server after, compress multiple block of this reception by the mode of parallel computation and multiple blocks of this compression are stored in the distributed file system of server.
2. file uploading method as claimed in claim 1, it is characterized in that, described dicing step comprises:
Receive the block worst case transmit time of user's definition, and utilize the size of each block after described transmission speed and the cutting of block worst case transmit time determination file.
3. file uploading method as claimed in claim 1, is characterized in that, also comprises before compression storing step:
Judge whether the capacity of described file is greater than a preset capacity, to determine whether the multiple blocks compressing this file.
4. file uploading method as claimed in claim 1, it is characterized in that, described dicing step also comprises:
Generate the block index being used for the plurality of block to be merged into a complete file.
5. file uploading method as claimed in claim 4, is characterized in that, describedly also comprises after compression storing step:
Receive the block index that client is sent to the file of server, and the block index of this file is stored in the database be connected with server.
6. a file uploading system, is applied in client, it is characterized in that, this system comprises:
Acquisition module, for obtaining the file needing to upload onto the server, and obtains the transmission speed between client and server;
Cutting module, for being multiple block according to described transmission speed by file cutting; And
Transport module, for by parallel transmission by described multiple encrypted communication in server, described server, after receiving multiple blocks of client transmissions, compresses multiple block of this reception by the mode of parallel computation and is stored in the distributed file system of server by multiple blocks of this compression.
7. file uploading system as claimed in claim 6, is characterized in that, file cutting is multiple block by following steps by described cutting module:
Receive the block worst case transmit time of user's definition, and utilize the size of each block after described transmission speed and the cutting of block worst case transmit time determination file.
8. a file uploading system, is applied in server, it is characterized in that, this system comprises:
Compression memory module, for receiving multiple blocks of client transmissions, described multiple block is by client according to the transmission speed between server, will the file uploaded onto the server be needed to carry out cutting, and transfers to server by parallel transmission;
Described compression memory module, also for being compressed multiple block of this reception by the mode of parallel computation and being stored in the distributed file system of server by multiple blocks of this compression.
9. file uploading system as claimed in claim 8, is characterized in that, described compression memory module also for, judge whether the capacity of described file is greater than a preset capacity, to determine whether the multiple blocks compressing this file.
10. file uploading system as claimed in claim 8, is characterized in that, described compression memory module also for, generate the block index being used for the plurality of block to be merged into a complete file.
11. file uploading systems as claimed in claim 10, is characterized in that, described compression memory module also for, receive client and be sent to the block index of the file of server, and the block index of this file is stored in the database be connected with server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310377622.1A CN104426942A (en) | 2013-08-27 | 2013-08-27 | File uploading method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310377622.1A CN104426942A (en) | 2013-08-27 | 2013-08-27 | File uploading method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104426942A true CN104426942A (en) | 2015-03-18 |
Family
ID=52974869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310377622.1A Pending CN104426942A (en) | 2013-08-27 | 2013-08-27 | File uploading method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104426942A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718538A (en) * | 2016-01-18 | 2016-06-29 | 中国科学院计算技术研究所 | Adaptive compression method and system for distributed file system |
| CN106682511A (en) * | 2016-10-31 | 2017-05-17 | 腾讯科技（深圳）有限公司 | Suspected virus file collection method and device |
| CN106790511A (en) * | 2016-12-16 | 2017-05-31 | 酷伴科技（北京）有限责任公司 | A kind of data uploading method and system |
| CN110266814A (en) * | 2019-07-09 | 2019-09-20 | 联想(北京)有限公司 | Transmission method and transmitting device |
| CN112583743A (en) * | 2020-09-28 | 2021-03-30 | 京信数据科技有限公司 | Distributed file exchange method and device |
| CN114448969A (en) * | 2021-12-27 | 2022-05-06 | 天翼云科技有限公司 | Data uploading and storing method and device, computer equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7193972B1 (en) * | 2000-10-06 | 2007-03-20 | Mitsubishi Denki Kabushiki Kaisha | Data transmission method and data transmitter |
| CN101605148A (en) * | 2009-05-21 | 2009-12-16 | 何吴迪 | The framework method of the parallel system of cloud storage |
| CN101980509A (en) * | 2010-11-05 | 2011-02-23 | 北京云快线软件服务有限公司 | File fragment transmission method and device |
| CN102790771A (en) * | 2012-07-25 | 2012-11-21 | 山东中创软件商用中间件股份有限公司 | File transmission method and system |
| CN102906751A (en) * | 2012-07-25 | 2013-01-30 | 华为技术有限公司 | Method and device for data storage and data query |
| CN103077149A (en) * | 2013-01-09 | 2013-05-01 | 厦门市美亚柏科信息股份有限公司 | Method and system for transmitting data |
| CN103259797A (en) * | 2013-05-17 | 2013-08-21 | 中国联合网络通信集团有限公司 | Data file transmission method and platform |
-
2013
- 2013-08-27 CN CN201310377622.1A patent/CN104426942A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7193972B1 (en) * | 2000-10-06 | 2007-03-20 | Mitsubishi Denki Kabushiki Kaisha | Data transmission method and data transmitter |
| CN101605148A (en) * | 2009-05-21 | 2009-12-16 | 何吴迪 | The framework method of the parallel system of cloud storage |
| CN101980509A (en) * | 2010-11-05 | 2011-02-23 | 北京云快线软件服务有限公司 | File fragment transmission method and device |
| CN102790771A (en) * | 2012-07-25 | 2012-11-21 | 山东中创软件商用中间件股份有限公司 | File transmission method and system |
| CN102906751A (en) * | 2012-07-25 | 2013-01-30 | 华为技术有限公司 | Method and device for data storage and data query |
| CN103077149A (en) * | 2013-01-09 | 2013-05-01 | 厦门市美亚柏科信息股份有限公司 | Method and system for transmitting data |
| CN103259797A (en) * | 2013-05-17 | 2013-08-21 | 中国联合网络通信集团有限公司 | Data file transmission method and platform |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718538A (en) * | 2016-01-18 | 2016-06-29 | 中国科学院计算技术研究所 | Adaptive compression method and system for distributed file system |
| CN105718538B (en) * | 2016-01-18 | 2019-05-14 | 中国科学院计算技术研究所 | A kind of self-adapting compressing method and system of distributed file system |
| CN106682511A (en) * | 2016-10-31 | 2017-05-17 | 腾讯科技（深圳）有限公司 | Suspected virus file collection method and device |
| CN106682511B (en) * | 2016-10-31 | 2019-02-12 | 腾讯科技（深圳）有限公司 | The collection method and device of doubtful virus document |
| CN106790511A (en) * | 2016-12-16 | 2017-05-31 | 酷伴科技（北京）有限责任公司 | A kind of data uploading method and system |
| CN110266814A (en) * | 2019-07-09 | 2019-09-20 | 联想(北京)有限公司 | Transmission method and transmitting device |
| CN112583743A (en) * | 2020-09-28 | 2021-03-30 | 京信数据科技有限公司 | Distributed file exchange method and device |
| CN114448969A (en) * | 2021-12-27 | 2022-05-06 | 天翼云科技有限公司 | Data uploading and storing method and device, computer equipment and storage medium |
| CN114448969B (en) * | 2021-12-27 | 2023-06-23 | 天翼云科技有限公司 | Data uploading and storing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104426942A (en) | File uploading method and system | |
| CN103873507A (en) | Data block uploading and storing system and method | |
| CN103024081B (en) | Be applicable to the terminal scheduling method of the point-to-point communication of effective guarantee communication system | |
| CN103312544B (en) | A kind of control terminal reports the method, apparatus and system of journal file | |
| CN103279351B (en) | A kind of method of task scheduling and device | |
| CN109981744B (en) | Data distribution method and device, storage medium and electronic equipment | |
| CN108334557B (en) | Aggregated data analysis method and device, storage medium and electronic equipment | |
| CN107743130B (en) | Fingerprint matching method, device and system | |
| CN105183371A (en) | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus | |
| US20160004455A1 (en) | Method of controlling splitting of data, and system | |
| CN110109899B (en) | Internet of things data filling method, device and system | |
| CN103095824A (en) | File uploading control method and system | |
| CN110324681B (en) | Slicing method, device, storage medium and transmission system of multimedia data | |
| RU2632420C1 (en) | Method of data processing and device used for application of terminal | |
| CN102882960A (en) | Method and device for transmitting resource files | |
| CN105357297A (en) | Data caching system and method | |
| CN103856521A (en) | File uploading system and method | |
| Lawson et al. | Data quality and energy management tradeoffs in sensor service clouds | |
| WO2021147319A1 (en) | Data processing method, apparatus, device, and medium | |
| CN102685222B (en) | A kind of cloud SRM device for electric power system | |
| CN103068052A (en) | Dynamic configuration method and system of resources and portal server | |
| CN110891087B (en) | Log transmission method and device, electronic equipment and storage medium | |
| CN102377580B (en) | The method for uploading of performance data and equipment | |
| CN105357060A (en) | Method, device and system for obtaining network equipment performance data | |
| CN108989088B (en) | Log uploading method and communication equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150318 |
|
| WD01 | Invention patent application deemed withdrawn after publication |