CN114338122A - Efficient and safe mass file transmission method - Google Patents
Efficient and safe mass file transmission method Download PDFInfo
- Publication number
- CN114338122A CN114338122A CN202111591436.9A CN202111591436A CN114338122A CN 114338122 A CN114338122 A CN 114338122A CN 202111591436 A CN202111591436 A CN 202111591436A CN 114338122 A CN114338122 A CN 114338122A
- Authority
- CN
- China
- Prior art keywords
- file
- transmission
- server
- client
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-efficiency safe mass file transmission method, relates to the technical field of communication, and solves the technical problems of low transmission efficiency and poor safety of mass files, and the method comprises the following steps: deploying a client process on a source file server, deploying a server process on a target file server, and establishing connection between the client and the server through a high-performance transmission engine; the client side preprocesses the file according to the size of the file; the client side carries out primary encryption processing on the file content; sending the file through a high-performance transmission engine, and carrying out secondary encryption processing on the content of the file in the transmission process; and after receiving the file sent by the client, the server performs HASH verification to confirm whether the file is successfully transmitted or not and decrypts the file. The invention sets the maximum file volume for one-time transmission according to the bandwidth, combines or splits the files and then sends the files through the high-performance transmission engine, encrypts the files once and encrypts the files twice in the transmission process, thereby improving the file transmission efficiency and safety.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a high-efficiency and safe mass file transmission method.
Background
With the development of cloud computing, big data and the Internet of things, the data volume owned by enterprises is exponentially and rapidly increased, and a large amount of files are generated. Due to the requirements of optimizing the business process, reducing the cost and improving the efficiency, the original mass files which are rarely used become treasury from which enterprises mine the business value.
How to use the massive files conveniently, uniformly manage and control the massive files and extract more commercial values from the massive files, not only provides higher requirements for data transmission capability of enterprise IT infrastructure, but also is a problem which needs to be solved for efficient operation of enterprises.
Because no special mass file transmission tool is available, enterprises generally use FTP, network disks and removable hard disks for mass file uploading and downloading. However, these methods have many disadvantages, which results in that enterprises cannot effectively utilize massive files. FTP, network disk, portable hard disk file are as the transmission tool of magnanimity file, the main problem that exists: 1. the transmission performance is low, and the transmission management overhead of the FTP and the network disk is fixed when each file is transmitted, so that the transmission management overhead ratio is greatly increased when massive files are transmitted, and the transmission performance is low; 2. the security problems of data leakage, cracking, monitoring and the like exist; 3. the transmission reliability is poor, and file loss or errors often occur during the transmission of mass files; 4. under the condition that the number of files is huge, FTP and a network disk cannot be transmitted at all, and the files are difficult to move; 5. the transmission file quantity is large, the transmission is slow, the transmission period is long, and the service requirement cannot be met.
In order to get a competitive lead from the market competition of the immense change, the enterprise demands digital transformation more and more strongly, and the passive digital transformation is changed into the active digital transformation. In the digital transformation process of enterprises, the safe and efficient transmission of massive files is a problem which has to be considered by the enterprises.
Disclosure of Invention
The invention aims to solve the technical problem of the prior art, and aims to provide a high-efficiency and safe mass file transmission method with high transmission efficiency and good safety.
The technical scheme of the invention is as follows: an efficient and safe mass file transmission method comprises the following steps:
deploying a client process on a source file server, deploying a server process on a target file server, and establishing connection between the client and the server through a high-performance transmission engine;
the client side preprocesses the file according to the size of the file;
the client encrypts the file content once;
the client sends the file to the server through the high-performance transmission engine, and secondary encryption processing is carried out on the content of the file in the transmission process;
and after receiving the file sent by the client, the server performs HASH verification to confirm whether the file is successfully transmitted or not and decrypts the file.
As a further improvement, the high-performance transmission engine comprises an intelligent compression component, an ultra-high speed data transmission protocol and an intelligent feedback component;
the intelligent compression assembly is internally provided with a plurality of compression algorithms, and can dynamically select the optimal compression algorithm to compress the file according to the file type to prepare for file transmission;
the principle of the ultra-high speed data transmission protocol for transmitting data comprises the following steps:
based on UDP protocol, sending by a timer;
the congestion control algorithm can realize that the bandwidth is quickly increased and seized in the slow start stage, and gradually reduces the increasing speed when the saturation is approached and the increasing speed tends to be stable;
the intelligent feedback component is used for feeding back packet loss information and only supplementing transmission aiming at real packet loss.
Furthermore, the high-performance transmission engine further comprises an automatic intelligent optimization component, the automatic intelligent optimization component automatically calculates the bandwidth size according to the bandwidth information, dynamically adjusts the maximum file size of one-time transmission according to the bandwidth size, and uses the network bandwidth to the maximum extent.
Further, the operation step of the high-performance transmission engine comprises:
a server side operation step, wherein the server side binds a certain local port when starting, and registers a NioServer SocketChannel of the server side to a selector of a certain boss NioEventLoop; the service end comprises 1 boss NioEventLoopgroup and 1 worker NioEventLoopgroup, the NioEventLoopgroup comprises a plurality of event loops NioEventLoop, and each NioEventLoop comprises 1 selector and 1 event loop thread;
and the client side runs, is connected to the server side when being started, establishes a NioSecketChannel and registers the NioEventLoop selector to a NioEventLoop selector, and only contains 1 NioEventLoopgroup.
Further, each task executed by the boss NioEventLoop loop comprises 3 steps:
step 1, polling an accept event;
step 2, an io task, namely an accept event, is processed, the connection is established with the client, NioSectChannel is generated, and the NioSectChannel is registered on a selector of a worker NioEventLoop;
and step 3, processing the tasks in the task queue, runAllTasks, wherein the tasks in the task queue comprise tasks which are executed by a user calling eventloop, execute or schedule or tasks which are submitted to the eventloop by other threads.
Further, each worker NioEventLoop loop executes a task comprising 3 steps:
step 1, polling read and write events;
step 2, the io tasks, namely read and write events, are processed when the NioSectChannel readable and writable event occurs;
and 3, processing a task in the task queue, runAllTasks.
Further, the task executed by the NioEventLoopgroup loop of the client comprises 3 steps:
step 1, polling connect, read and write events;
step 2, processing io tasks, namely connect, read and write events, and processing when NioSecketChannel connection is established, and readable and writable events occur;
and 3, processing a non-io task, runAllTasks.
Further, preprocessing the file comprises merging processing and splitting processing;
when the file is far smaller than the maximum value of the file transmitted each time, combining a plurality of small files together by using a virtual splicing technology for transmission;
when the file is larger than the maximum value of the file transmitted each time, the file is divided into a plurality of file segments to be transmitted.
Further, the first encryption process is to encrypt the file content by using an AES-256 encryption algorithm, and the second encryption process is to encrypt the file content by using SSL.
Furthermore, the state of the data transmission task is monitored in real time in the transmission process, and the task priority is suspended, stopped or adjusted according to the service requirement.
Advantageous effects
Compared with the prior art, the invention has the advantages that:
the invention sets the maximum file size for one-time transmission according to the bandwidth size, uses the network bandwidth to the maximum extent, and sends the file through the high-performance transmission engine after carrying out merging or splitting pretreatment on the file, thereby improving the file transmission efficiency.
Drawings
FIG. 1 is an architecture diagram of a client and a server deployed in the present invention;
FIG. 2 is a schematic diagram of merging files according to the present invention;
FIG. 3 is a diagram illustrating splitting a file according to the present invention.
Detailed Description
The invention will be further described with reference to specific embodiments shown in the drawings.
Referring to fig. 1 to 3, a method for efficiently and safely transmitting a mass file includes:
deploying a client process on a source file server, deploying a server process on a target file server, and establishing connection between the client and the server through a high-performance transmission engine; when the connection is established, the user account and the identity are managed and verified;
the client side preprocesses the file according to the size of the file;
the client side carries out primary encryption processing on the file content;
the client sends the file to the server through the high-performance transmission engine, and secondary encryption processing is carried out on the file content in the transmission process;
and after receiving the file sent by the client, the server performs HASH verification to confirm whether the file is successfully transmitted or not and decrypts the file.
The high-performance transmission engine comprises an intelligent compression component, an ultra-high-speed data transmission protocol and an intelligent feedback component. The intelligent compression component is internally provided with a plurality of compression algorithms, can dynamically select the optimal compression algorithm to compress the file according to the file type, for example, different compression algorithms are adopted for compressing video files, audio files and document files, so that the transmitted file is as small as possible, and the preparation is made for file transmission.
The principle of the ultra-high speed data transmission protocol for transmitting data comprises the following steps: based on UDP protocol, the timer sends, without ACK like TCP, the next round of sending is started; the congestion control algorithm can realize the rapid increase of the occupied bandwidth in the slow start stage, gradually reduce the increase speed when approaching saturation and tend to be stable. The algorithm for processing packet loss and the tolerance of the ultra-high speed data transmission protocol to a noise link enable the ultra-high speed data transmission protocol to be more stable than the traditional TCP protocol in an environment with larger network fluctuation. The intelligent feedback component is used for feeding back packet loss information, the server side detects packet loss and feeds back the packet information needing to be retransmitted to the sending end, and the sending end only supplements transmission aiming at real packet loss by means of a retransmission mechanism of an ultra-high speed transmission protocol, so that redundant transmission and bandwidth waste are avoided, retransmission efficiency is high, and efficient and reliable transmission is achieved.
The high-performance transmission engine also comprises an automatic intelligent optimization component, the automatic intelligent optimization component automatically calculates the bandwidth size according to the bandwidth information, the maximum file size in one-time transmission is dynamically adjusted according to the bandwidth size, and the network bandwidth is used to the maximum extent. The existing bandwidth is fully utilized, the influences of network delay, packet loss and the like are effectively reduced, and ultra-long-distance, cross-region large file and mass small file high-speed transmission is realized.
The operation steps of the high-performance transmission engine comprise:
a server side operation step, wherein the server side binds a certain local port when starting, and registers a NioServer SocketChannel of the server side to a selector of a certain boss NioEventLoop; the server comprises 1 boss NioEventLoopgroup and 1 worker NioEventLoopgroup, wherein the NioEventLoopgroup comprises a plurality of event loops NioEventLoop, and each NioEventLoop comprises 1 selector and 1 event loop thread.
The task executed by each boss NioEventLoop loop comprises 3 steps:
step 1, polling an accept event;
step 2, an io task, namely an accept event, is processed, the connection is established with the client, NioSectChannel is generated, and the NioSectChannel is registered on a selector of a worker NioEventLoop;
and step 3, processing the tasks in the task queue, runAllTasks, wherein the tasks in the task queue comprise tasks which are executed by a user calling eventloop, execute or schedule or tasks which are submitted to the eventloop by other threads.
The task executed by each worker NioEventLoop loop comprises 3 steps:
step 1, polling read and write events;
step 2, the io tasks, namely read and write events, are processed when the NioSectChannel readable and writable event occurs;
and 3, processing a task in the task queue, runAllTasks.
And the client side runs, is connected to the server side when being started, establishes a NioSecketChannel and registers the NioEventLoop selector to a NioEventLoop selector, and only contains 1 NioEventLoopgroup.
The task executed by the NioEventLoopgroup loop of the client comprises 3 steps:
step 1, polling connect, read and write events;
step 2, processing io tasks, namely connect, read and write events, and processing when NioSecketChannel connection is established, and readable and writable events occur;
and 3, processing a non-io task, runAllTasks.
The file preprocessing in the invention comprises merging processing and splitting processing. When the file is far smaller than the maximum value of the file transmitted each time, combining a plurality of small files together by using a virtual splicing technology for transmission; and recording the splicing information, and after receiving the corresponding file, the subsequent server can restore the file according to the splicing virtual information, so that the number of file transmission times can be effectively reduced, as shown in fig. 2.
When the file is larger than the maximum value of the file transmitted each time, the file is split into a plurality of file segments to be transmitted, corresponding virtual file segment information is transmitted to the server, and the subsequent server restores the file according to the virtual file segment information after receiving the file, as shown in fig. 3.
In the embodiment, the primary encryption processing is to encrypt the file content by adopting an AES-256 encryption algorithm, and the secondary encryption processing is to use SSL encryption, so that the data security is effectively ensured.
The data transmission task state is monitored in real time in the file transmission process, the overall progress of file transmission is monitored, and a user can conveniently master the file transmission condition; the instant speed of file transmission is monitored, so that a user can conveniently master the speed of file transmission; the time consumption of file transmission is monitored, so that a user can conveniently master the file transmission time; the abnormal condition of file transmission is monitored, so that a user can conveniently master the abnormal condition of file transmission; the supervision file transmits the state of each node, so that a user can conveniently master the state of each node.
And pausing, stopping or adjusting the task priority of the transmission task according to the service requirement in the file transmission process. In the process of file transmission, if the transmission task needs to be temporarily suspended, the transmission of files is suspended; in the file transmission process, if the transmission task needs to be terminated and the file transmission is not finished, stopping transmitting the file; in the process of file transmission, due to service requirements, the priority of a certain task is improved so as to transmit the corresponding file preferentially.
The file transmission process also comprises an operation step of intelligent breakpoint continuous transmission, and the method is mainly used for ensuring the reliable transmission of the files. Firstly, recording state information in the file transmission process; setting a periodic timing task, and checking the file transmission result at regular time; in the transmission process, when the task state is changed when abnormal transmission is found, in order to ensure the stability and the integrity of data transmission, the transmission is ensured to be capable of effectively ensuring the data transmission when the conditions of network faults, abnormal transmission and the like occur in the transmission process through breakpoint continuous transmission, error retransmission, multiple file verification and an intelligent synchronization mechanism, and meanwhile, the output time is shortened, and the work load is lightened.
The above is only a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that several variations and modifications can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the utility of the patent.
Claims (10)
1. A high-efficiency safe mass file transmission method is characterized by comprising the following steps:
deploying a client process on a source file server, deploying a server process on a target file server, and establishing connection between the client and the server through a high-performance transmission engine;
the client side preprocesses the file according to the size of the file;
the client encrypts the file content once;
the client sends the file to the server through the high-performance transmission engine, and secondary encryption processing is carried out on the content of the file in the transmission process;
and after receiving the file sent by the client, the server performs HASH verification to confirm whether the file is successfully transmitted or not and decrypts the file.
2. An efficient and safe mass file transmission method as claimed in claim 1, wherein the high performance transmission engine comprises an intelligent compression component, an ultra-high speed data transmission protocol, an intelligent feedback component;
the intelligent compression assembly is internally provided with a plurality of compression algorithms, and can dynamically select the optimal compression algorithm to compress the file according to the file type to prepare for file transmission;
the principle of the ultra-high speed data transmission protocol for transmitting data comprises the following steps:
based on UDP protocol, sending by a timer;
the congestion control algorithm can realize that the bandwidth is quickly increased and seized in the slow start stage, and gradually reduces the increasing speed when the saturation is approached and the increasing speed tends to be stable;
the intelligent feedback component is used for feeding back packet loss information and only supplementing transmission aiming at real packet loss.
3. The method according to claim 1, wherein the high performance transport engine further comprises an automatic intelligent optimization component, the automatic intelligent optimization component automatically calculates the bandwidth size according to the bandwidth information, dynamically adjusts the maximum file size for one-time transport according to the bandwidth size, and maximally uses the network bandwidth.
4. The method for efficient and secure transfer of mass files according to claim 1, wherein the step of operating the high performance transfer engine comprises:
a server side operation step, wherein the server side binds a certain local port when starting, and registers a NioServer SocketChannel of the server side to a selector of a certain boss NioEventLoop; the service end comprises 1 boss NioEventLoopgroup and 1 worker NioEventLoopgroup, the NioEventLoopgroup comprises a plurality of event loops NioEventLoop, and each NioEventLoop comprises 1 selector and 1 event loop thread;
and the client side runs, is connected to the server side when being started, establishes a NioSecketChannel and registers the NioEventLoop selector to a NioEventLoop selector, and only contains 1 NioEventLoopgroup.
5. The method for efficient and safe mass file transmission according to claim 4, wherein the task executed circularly by each boss NioEventLoop comprises 3 steps:
step 1, polling an accept event;
step 2, an io task, namely an accept event, is processed, the connection is established with the client, NioSectChannel is generated, and the NioSectChannel is registered on a selector of a worker NioEventLoop;
and step 3, processing the tasks in the task queue, runAllTasks, wherein the tasks in the task queue comprise tasks which are executed by a user calling eventloop, execute or schedule or tasks which are submitted to the eventloop by other threads.
6. The efficient and safe mass file transmission method according to claim 4, wherein the task executed circularly by each worker NioEventLoop comprises 3 steps:
step 1, polling read and write events;
step 2, the io tasks, namely read and write events, are processed when the NioSectChannel readable and writable event occurs;
and 3, processing a task in the task queue, runAllTasks.
7. The method for transmitting the high-efficiency and safe mass files according to claim 4, wherein the task executed circularly by the NioEventLoopgroup of the client comprises 3 steps:
step 1, polling connect, read and write events;
step 2, processing io tasks, namely connect, read and write events, and processing when NioSecketChannel connection is established, and readable and writable events occur;
and 3, processing a non-io task, runAllTasks.
8. The method for transmitting the high-efficiency safe mass files according to claim 1, wherein the file preprocessing comprises merging processing and splitting processing;
when the file is far smaller than the maximum value of the file transmitted each time, combining a plurality of small files together by using a virtual splicing technology for transmission;
when the file is larger than the maximum value of the file transmitted each time, the file is divided into a plurality of file segments to be transmitted.
9. An efficient and safe mass file transmission method as claimed in claim 1, wherein the primary encryption process is to encrypt the file contents by using AES-256 encryption algorithm, and the secondary encryption process is to use SSL encryption.
10. The method for transmitting the high-efficiency and safe mass files according to claim 1, wherein the state of the data transmission task is monitored in real time in the transmission process, and the task is suspended, stopped or the task priority is adjusted according to the service requirement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111591436.9A CN114338122A (en) | 2021-12-23 | 2021-12-23 | Efficient and safe mass file transmission method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111591436.9A CN114338122A (en) | 2021-12-23 | 2021-12-23 | Efficient and safe mass file transmission method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114338122A true CN114338122A (en) | 2022-04-12 |
Family
ID=81053633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111591436.9A Pending CN114338122A (en) | 2021-12-23 | 2021-12-23 | Efficient and safe mass file transmission method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114338122A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118526A (en) * | 2022-08-23 | 2022-09-27 | 北京润尼尔网络科技有限公司 | VR device data migration method |
| CN116455527A (en) * | 2023-06-16 | 2023-07-18 | 深圳华锐分布式技术股份有限公司 | Message retransmission method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001054370A2 (en) * | 2000-01-24 | 2001-07-26 | The University Of Manitoba | Method and system for segmented file transfer |
| CN103986696A (en) * | 2014-04-24 | 2014-08-13 | 华为技术有限公司 | Multimedia file transmission device and method |
| CN110545329A (en) * | 2019-09-27 | 2019-12-06 | 杭州海潮信息科技有限公司 | Method for improving FTP file transmission speed |
| CN111767258A (en) * | 2020-06-30 | 2020-10-13 | 深圳前海微众银行股份有限公司 | File compression method, device, equipment and storage medium applied to mass files |
| CN112615899A (en) * | 2020-11-25 | 2021-04-06 | 北京中电普华信息技术有限公司 | Large file transmission method, device and system |
-
2021
- 2021-12-23 CN CN202111591436.9A patent/CN114338122A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001054370A2 (en) * | 2000-01-24 | 2001-07-26 | The University Of Manitoba | Method and system for segmented file transfer |
| CN103986696A (en) * | 2014-04-24 | 2014-08-13 | 华为技术有限公司 | Multimedia file transmission device and method |
| CN110545329A (en) * | 2019-09-27 | 2019-12-06 | 杭州海潮信息科技有限公司 | Method for improving FTP file transmission speed |
| CN111767258A (en) * | 2020-06-30 | 2020-10-13 | 深圳前海微众银行股份有限公司 | File compression method, device, equipment and storage medium applied to mass files |
| CN112615899A (en) * | 2020-11-25 | 2021-04-06 | 北京中电普华信息技术有限公司 | Large file transmission method, device and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118526A (en) * | 2022-08-23 | 2022-09-27 | 北京润尼尔网络科技有限公司 | VR device data migration method |
| CN115118526B (en) * | 2022-08-23 | 2022-11-25 | 北京润尼尔网络科技有限公司 | VR device data migration method |
| CN116455527A (en) * | 2023-06-16 | 2023-07-18 | 深圳华锐分布式技术股份有限公司 | Message retransmission method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114338122A (en) | Efficient and safe mass file transmission method | |
| US10069766B2 (en) | Accelerated data transfer using thread pool for parallel operations | |
| CN109309730B (en) | Credible file transmission method and system | |
| CN102833337B (en) | A kind of ftp file is uploaded, method for down loading and device | |
| US6665729B2 (en) | Data transmission utilizing pre-emptive acknowledgements with transaction-oriented protocols | |
| US8898311B2 (en) | Data communication method and information processing device | |
| US8954525B2 (en) | Method and apparatus of performing remote computer file exchange | |
| CN104216806B (en) | Catching and transmission method and device thereof of a kind of file system serializing Operation Log | |
| US20110246763A1 (en) | Parallel method, machine, and computer program product for data transmission and reception over a network | |
| US11303737B2 (en) | Method and device for data transmission | |
| WO2023216424A1 (en) | Data link service processing system and method for networked encrypted transmission | |
| WO2020233054A1 (en) | Blockchain-based block generation method, apparatus and device, and storage medium | |
| US20060265511A1 (en) | Apparatus and method for efficiently and securely transferring files over a communications network | |
| CN112910967A (en) | Network transmission of large files in unstable network environments | |
| CN104580158A (en) | Distributed platform file and content distribution method and distributed platform file and content distribution system | |
| CN114338651A (en) | File transmission method and device, electronic equipment and readable storage medium | |
| US9240952B2 (en) | System and method for communication between networked applications | |
| CN101771718A (en) | Clipboard synchronous method and system | |
| CN111355785A (en) | FTP (File transfer protocol) secure transmission system and method based on CMSP (China Mobile protocol service) | |
| US8719902B2 (en) | Secure communication device, secure communication method, and program | |
| CN105471718A (en) | Realization method of full duplex message queue | |
| GB2537459A (en) | Transmitting data | |
| JP7075077B2 (en) | Backup server, backup method, program, storage system | |
| KR20040047207A (en) | Backup system with load balancer for data backup or extracting and method for data backup using the same | |
| CN106162731A (en) | Industrial field data transmission method based on 3G wireless network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |