CN111953784B

CN111953784B - Asynchronous communication framework-based file transmission method, device and system

Info

Publication number: CN111953784B
Application number: CN202010817472.1A
Authority: CN
Inventors: 徐平; 丁文定; 张雯; 莫享尔
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-08-14
Filing date: 2020-08-14
Publication date: 2023-01-06
Anticipated expiration: 2040-08-14
Also published as: CN111953784A

Abstract

The invention discloses a file transmission method, a device and a system based on an asynchronous communication frame, belonging to the field of big data, wherein the method comprises the following steps: responding to the starting operation of the node, sending the node information of the node to a console through a node management module so that the console can conveniently verify the node information, and the node management module realizes the interaction between the node and the console through remote method calling; responding to the successful verification of the node information, receiving system information and transmission task information of a node of a control console through a node management module, wherein the system information is used for monitoring transmission operation; and establishing communication connection between the node and the target node based on the asynchronous communication framework and the system information, and transmitting a corresponding file with the target node on the established communication connection according to the transmission task information. The invention can realize high transmission rate of the file and improve the file transmission performance.

Description

Asynchronous communication framework-based file transmission method, device and system

Technical Field

The invention relates to the technical field of communication, in particular to a file transmission method, a device and a system based on an asynchronous communication framework.

Background

At present, the amount of data to be exchanged between banking application systems is huge, some commercial banks use FTP (File Transfer Protocol) Protocol-based software to perform point-to-point File Transfer, and some commercial banks also use platform-based File Transfer systems, such as a batch File Transfer platform built on TongGTP (a File Transfer platform) products of the eastern square company.

However, these technologies have the disadvantages of low processing performance, high transmission failure rate, and the like in a scenario of transmitting a large number of small files.

Disclosure of Invention

In view of the above, the present invention provides a method, an apparatus, and a system for file transfer based on an asynchronous communication framework to solve at least one of the above-mentioned problems.

According to a first aspect of the present invention, there is provided a file transfer method based on an asynchronous communication framework, the method comprising: responding to the starting operation of the node, sending the node information of the node to a console through a node management module so that the console can conveniently verify the node information, and the node management module realizes the interaction between the node and the console through remote method calling; responding to the successful verification of the node information, receiving system information and transmission task information of a node to which the control console belongs through the node management module, wherein the system information is used for monitoring transmission operation; and establishing communication connection between the node and the target node based on the asynchronous communication frame and the system information, and transmitting a corresponding file with the target node on the established communication connection according to the transmission task information.

According to a second aspect of the present invention, there is provided a file transfer apparatus based on an asynchronous communication framework, the apparatus comprising: the information sending unit is used for responding to the starting operation of the node to which the node belongs and sending the node information of the node to a console through a node management module so as to be convenient for the console to carry out verification operation on the node information, and the node management module realizes the interaction between the node to which the node belongs and the console through remote method calling; the information receiving unit is used for responding to the successful verification of the node information and receiving system information and transmission task information of the node to which the control console belongs through the node management module, wherein the system information is used for monitoring transmission operation; the connection establishing unit is used for establishing communication connection between the node and the target node based on the asynchronous communication frame and the system information; and the file transmission unit is used for transmitting a corresponding file with the target node on the established communication connection according to the transmission task information.

According to a third aspect of the present invention, there is provided a file transfer system based on an asynchronous communication framework, the system comprising: the file receiving device is positioned at a target node, and the file receiving device and the file transmitting device transmit files through the established communication connection.

According to a fourth aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the program.

According to a fifth aspect of the invention, the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the above-mentioned method.

According to the technical scheme, when the node is started, the node information is sent to the console through the node management module, the console verifies the node information, when the node information is verified successfully, the node management module receives the system information and the transmission task information from the console, then the communication connection between the node information and the target node is established based on the asynchronous communication frame and the system information, and the corresponding file is transmitted to the target node on the established communication connection according to the transmission task information.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a block diagram of a file transfer system based on an asynchronous communication framework according to an embodiment of the present invention;

fig. 2 is a block diagram of the structure of the file transfer device 2 according to the embodiment of the present invention;

fig. 3 is a detailed structural block diagram of the file transfer apparatus 2 according to the embodiment of the present invention;

fig. 4 is a block diagram of the structure of the console 1 according to the embodiment of the present invention;

fig. 5 is a block diagram of the structure of the file receiving apparatus 3 according to the embodiment of the present invention;

FIG. 6 is an exemplary architecture diagram of a Netty framework based file transfer system according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a transfer flow of a Netty framework based file according to an embodiment of the invention;

fig. 8 is a block diagram of a Client sender according to an embodiment of the present invention;

fig. 9 is a first schematic diagram illustrating a file transmission flow of a Client sender according to an embodiment of the present invention;

fig. 10 is a schematic diagram illustrating a file transmission flow of a Client sender according to an embodiment of the present invention;

fig. 11 is a schematic diagram of a file transmission flow of a Client sending end and a Server receiving end according to an embodiment of the present invention;

FIG. 12 is a schematic diagram of a file transfer state according to an embodiment of the invention;

FIG. 13 is a flowchart of a file transfer method based on an asynchronous communication framework according to an embodiment of the present invention;

fig. 14 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In view of the problems of low processing performance, high transmission failure rate and the like of the existing technology for transmitting a large number of small files, the embodiment of the invention provides a file transmission scheme based on an asynchronous communication framework (such as the asynchronous communication framework Netty), which belongs to the field of big data.

Fig. 1 is a block diagram of a file transfer system based on an asynchronous communication framework according to an embodiment of the present invention, as shown in fig. 1, the system includes: the system comprises a console 1, a file transmission device 2 (namely a file sending end) and a file receiving device 3 (namely a file receiving end), wherein the console 1 manages file tasks transmitted by the file transmission device 2; the file receiving device and the file transmitting device transmit files through the communication connection established between the two devices, the file transmitting device 2 can be located at one transmitting node, and the file receiving device 3 can be located at another transmitting node (also called target node).

For a better understanding of the embodiment of the present invention, the document transportation apparatus 2 will be described below.

Fig. 2 is a block diagram showing the configuration of the file transfer apparatus 2, and as shown in fig. 2, the file transfer apparatus 2 includes: an information sending unit 21, an information receiving unit 22, a connection establishing unit 23, and a file transfer unit 24, wherein:

the information sending unit 21 is configured to respond to a node start operation, send node information of the node to the console 1 through a node management module, so that the console performs a verification operation on the node information, and the node management module realizes interaction between the node and the console through remote method invocation.

And the information receiving unit 22 is configured to receive, through the node management module, system information and transmission task information of a node to which the console belongs in response to the node information being successfully verified, where the system information is used to monitor a transmission operation.

The system information herein may include: node ID, file transmission failure retransmission threshold (i.e., number of times), transmission timeout threshold (i.e., time), scanning period, monitoring alarm, etc. parameters related to file transmission. In practical operation, the system information may further include: the size of a thread pool divided by the granularity of a target node (i.e., a receiving end node), the size of a transmission thread pool divided by the granularity of a file within a single target node, and the like.

A connection establishing unit 23, configured to establish a communication connection between the node and the target node based on the asynchronous communication framework and the system information.

And the file transmission unit 24 is configured to transmit a corresponding file with the target node on the established communication connection according to the transmission task information.

When the node is started, the information sending unit 21 sends the node information to the console through the node management module, the console performs verification operation on the node information, when the node information is successfully verified, the information receiving unit 22 receives system information and transmission task information from the console through the node management module, then the connection establishing unit 23 establishes communication connection with a target node based on an asynchronous communication frame and the system information, and the file transmission unit 24 transmits a corresponding file with the target node on the established communication connection according to the transmission task information.

In a specific implementation, the system information may be represented by an exemplary system information table as shown in table 1 below:

TABLE 1

The transmission task information may be represented by an exemplary transmission task information table as shown in table 2 below, in which the TaskInfo is stored using a convurrenthashmap.

TABLE 2

In a specific implementation, the connection establishing unit 23 may include: a request initiating module and a connection establishing module, wherein:

a request initiating module, configured to initiate a connection establishment request to the target node based on an asynchronous communication framework and the system information;

and the connection establishing module is used for successfully establishing the communication connection between the node and the target node when receiving the connection establishing response of the target node.

In the embodiment of the present invention, the target node may be plural.

The above-mentioned file transfer unit 24 includes: the device comprises a file to be transmitted acquisition module, a queue processing module and a file transmission module, wherein:

the file to be transmitted acquiring module is used for acquiring a file to be transmitted in the transmission task information and adding the file to be transmitted into a main task queue;

the queue processing module is used for adding the file to be transmitted into a subtask queue based on the target node, the subtask queue is used for indicating a transmission task of the file to be transmitted and the target node, and the main task queue and the subtask queue are associated based on key values;

and the file transmission module is used for transmitting the files to be transmitted in the subtask queue to a plurality of target nodes.

In one embodiment, as shown in fig. 3, the file transfer device 2 further includes: and a retransmission unit 25. When the file transmission with the target node fails over the established communication connection, the retransmission unit 25 may perform retransmission processing on the file based on the file exception handling and retransmission mechanism set in advance. Thus, the failure rate of file transmission can be reduced.

In practical operation, the file transfer system may further include: and the interaction unit is used for realizing the interaction between the console and the file transmission device.

For further understanding of the embodiments of the present invention, the following description is continued with the console 1 and the file receiving apparatus 3 in the file transfer system.

As shown in fig. 4, the console 1 includes: a node verification unit 11, a node information sending unit 12 and a task management unit 13, wherein:

a node verifying unit 11, configured to perform a verifying operation on node information;

the node information sending unit 12 is configured to send system information and transmission task information to the node after the node information is successfully verified;

and the task management unit 13 is used for managing the task state in the file transmission device. The task state here may be a preparation state, a processing state, a stop state, and a stop state.

As shown in fig. 5, the document receiving apparatus 3 includes: a connection establishing unit 31 and a file receiving unit 32, wherein:

a connection establishing unit 31 for establishing a communication connection with the file transfer apparatus in accordance with a connection establishment request from the file transfer apparatus;

a file receiving unit 32, configured to receive the file from the file transmission device.

In practical operation, the units and the modules may be combined or may be singly arranged, and the present invention is not limited thereto.

In one embodiment, the asynchronous communication framework can be a high-performance asynchronous communication framework Netty, the framework realizes a service end/client end transmission platform of NIO (Non-blocking I/O), and by utilizing a thread pool mode and a zero copy mechanism provided by Netty and matching with an Lz4 compression algorithm and a custom codec, the high-concurrency and high-speed file transmission function can be realized, and high-speed reliable transmission of large-batch files is realized.

In actual operation, the Netty framework-based file transfer system can be applied to two service scenes, namely file transfer between application systems in an enterprise and file transfer between a headquarter and a branch office. The system can support more than 1000 transmission nodes to simultaneously and concurrently carry out many-to-many transmission processing, the daily processing file is more than 100000, and the maximum processing total amount is more than 10T. With this system, under the gigabit network bandwidth condition, when the data amount is K (kilo) class, the transmission time is within 1 second, when the data amount is M (mega) class, the transmission time is within 6 seconds, and when the data amount is G (giga) class, the transmission time is within 20 seconds.

Fig. 6 is an exemplary architecture diagram of the Netty frame based file transfer system, which is described in detail below in conjunction with fig. 6.

As shown in fig. 6, the system includes: a front-end management console (shown in the figure as an ETRAN (a cross-platform transport tool) management console), a back-end transport system (i.e., transport node a and transport node B shown in the figure), and a node management module for interaction between the two.

The management console can manage the transmission nodes distributed on different servers, and comprises the following steps: starting and stopping nodes, displaying node states, verifying node information and issuing node system parameters; meanwhile, the tasks can be newly added, deleted and modified, and the tasks can be stopped and restarted.

The back-end transmission system transmits files by adopting a Server/Client mode based on a Netty framework. The Server side is a file receiving side (having the function of the file receiving apparatus 3), and the Client side is a file transmitting side (having the function of the file transmitting apparatus 2). Netty can isolate the network layer from the application layer, which only needs to be concerned about the business logic of file transmission and reception, while the handling and performance issues related to communication are handled by Netty.

The system further comprises: and the node management module (with the function of the interaction unit) realizes the interaction between the front-end management console and the back-end transmission system.

After the transmission node a or B is started, the node management module checks node information from the console by using Remote Method Invocation (RMI) technology to obtain node system information and task information, which may specifically refer to the flow shown in fig. 7, where as shown in fig. 7, the transmission node sends a node name, an IP, a port, and a version number in a node information table to the management console through the node management module to check node validity. After the verification is passed, the transmission node obtains the system information and the transmission task of the node from the management console through the node management module, then stores the system information into the system information table nodsyslnfo, and stores the obtained transmission task into the task information table TaskInfo.

The Client side (shown as ETRAN Client in FIG. 6) and the Server side (shown as ETRAN Server in FIG. 6) are described in detail below.

(I) Client terminal

As shown in fig. 8, the Client sender includes: a scanning module 81, a sending thread management module 82, a file transfer module 83, and a task management module 84, wherein: the scanning module scans and sends files to be sent in the directory at regular time, adds the scanned files into the task main table and the task sub table, and establishes respective sending queues according to target receiving nodes (namely target nodes) of the tasks; the sending thread management module monitors the sending queue list and maintains the sending thread pool resources; the file transmission module is responsible for specific file transmission, failure retransmission, transmission state management and the like; and the task management module maintains the task information in the task main table and the task sub table according to the task transmission state.

Fig. 9 is a schematic diagram of a file transmission flow of a Client sender, and as shown in fig. 9, a scanning thread (i.e., a scanning module 81) traverses a task information table TaskInfo, and if a task state is "Idle" or "Sending", scans a directory and a subdirectory corresponding to the task, and then adds the scanned files to a MainTask table and a sublatk task table, respectively. The MainTask is a main queue for sending tasks, and one file to be sent corresponds to one MainTask record.

The data structure of MainTask is shown in Table 3 below:

TABLE 3

A main task record may correspond to multiple sub task subtasks according to multiple target receiving nodes, and the sub task data structure is shown in table 4:

TABLE 4

SubTask is associated with MainTask by a Key value (Key value). Meanwhile, according to the target receiving node (node name) in the SubTask key value, a Queue is established for each receiving node, and the SubTask is added to the Queue. For example, the task1 corresponds to a sending directory D:/a, and the file receiving nodes are nodeA, nodeB and nodeC. And if the scanning thread scans the file bbb.txt to be sent under the path, adding a record of Key = task1| bbb.txt in the corresponding MainTask. Correspondingly, three sending subtasks are added in the SubTask, and the SubTask key values are respectively as follows:

Key1＝task1|bbb.txt@nodeA，

Key2＝task1|bbb.txt@nodeB，

Key3＝task1|bbb.txt@nodeC。

and then, inserting the subtasks into Queue _ nodeA, queue _ nodeB and Queue _ nodeC sending queues corresponding to the nodeA, nodeB and nodeC nodes respectively. This send queue list is the data structure on which the send thread management module depends.

In order to improve the transmission timeliness of the file transmitted to a plurality of receiving nodes, the transmitting thread management module 82 of the system designs a receiving node thread pool management mode, when one transmitting node needs to transmit the file to a plurality of receiving nodes, the system supports and transmits the file to different receiving nodes simultaneously, and the problem that the file receiving of other receiving nodes is influenced due to the blockage of one receiving node can be avoided. The sending thread management module obtains a sending thread from a thread pool (shown as a Netty sending thread pool in fig. 9) according to the number of core threads of the thread pool and a sending queue List < List > QueueN, and when a queue corresponding to the thread is empty, the sending thread management module indicates that the file sending is completed, ends the thread and recovers thread resources.

Fig. 10 is a schematic diagram of a file transmission flow of a Client sender, and as shown in fig. 10, the flow includes: and the scanning module scans the directory, and when a file to be sent exists in the directory, the information of the file to be sent is added into the MainTask table and the SubTask table. Firstly, a message header (header) is sent to a receiving end (namely, a receiving node), when the sending is successful, the receiving end is waited to feed back a receiving result, otherwise, the message header is retransmitted. When receiving the success of the feedback receiving message header of the receiving end, the file (file) is sent next, and then similarly, the receiving end is waited to feed back the receiving result, otherwise, the file is retransmitted. When receiving the feedback of the receiving end that the file is successfully received, the file is moved to the succ directory, and the corresponding tasks in the MainTask table and the SubTask table are deleted.

When either the message header or the file is retransmitted, the file is moved to the fail directory and the corresponding tasks in the MainTask table and the subttask table are deleted.

The file transfer module 83 of the Client sender is based on the Netty framework. Netty is an asynchronous, event-driven based web application framework that can be used to develop high performance servers and clients. When a network program is written in the related art, a socket is usually created at a client, and the client is connected to a server through the socket. The server creates a Thread for each socket. After the client initiates the call, the client waits for the completion of the processing of the server to continue the subsequent operation, which causes thread waiting. The number of the client requests is large, and the server needs to create a large number of processing threads, so that the problem of connection failure caused by insufficient thread resources is caused. Netty is a communication framework based on the NIO mechanism, in the NIO mechanism, a Socket Channel is created each time a client initiates a connection request, and is registered on a multiplexer (Selector), the Selector pays attention to an IO (Input/Output) read-write event of a server, and the client can continue subsequent work without waiting for the completion of the IO event. Thus, netty is more suitable for network communication in high concurrency scenarios.

In the embodiment of the present invention, a specific message interaction mechanism is provided, as shown in fig. 11. A sending node (shown as Client in the figure) firstly initiates a connection establishment request, and after receiving a connection establishment response of a receiving node (shown as Server in the figure), sends a message header, wherein the message header structure is as follows:

the definitions of the above message headers are shown in table 5:

numbering	Name of field	Means of
			1	destDir	Receiving a catalog
2	filename	Filename of file
			3	filesize	File size
4	duplicated	File overlay identification
			5	localip	Transmitting node IP
7	nodeName	Sending node names
			8	taskId	Sending task ID
9	taskName	Sending task names
			10	md5	MD5 check value

TABLE 5

And after receiving the message header receiving success response message, sending the file. And after receiving the complete file, the receiving node checks the file, calculates an MD5 value and checks the integrity of the file. And after the verification is passed, informing the sending node of the transmission result. And then, the sending node updates the sending task state according to the transmission result and closes the transmission channel.

In actual operation, a sender may create a thread pool of NioEventLoopGroup and a thread pool of boottrap, and a transmission Channel encoder is set as:

Lz4FrameEncoder

ByteArrayEncoder

ChunkedWriteHandler

MyReadTimeoutHandler

chunkFileClientHandler

the Lz4FrameEncoder is an Lz4 compression encoder, and the addition of the encoder can realize file compression and transmission, improve the transmission rate and save the network bandwidth. Lz4 is more than 3 times as fast as Gzip, etc. compression tools in compression speed, the encoder is suitable for fast transmission scenarios. The ByteArrayEncoder can solve the problem of packet sticking in the data transmission process. ChunkedWriteHandler provides a method for transmitting large files for Netty, and the method transmits the sending data according to the size of a designated transmission block without causing the risk of OutOfMemoryError (memory overflow). The MyReadTimeoutHandler is a timeout handler, which inherits a Netty read timeout control handler (an asynchronous handler) and is used for controlling corresponding processing when no data is read within a set time period, including releasing a connection and setting a task state as failure. The chunkFileClientHandler is used for completing message header and file sending and processing the feedback message of the receiving node.

In one embodiment, the following exception scenarios occur: the connection is failed to be established; capturing an exception in the sending process of the Client message head; after receiving the message header, the Server end processes the exception and sends a fail response message to the Client; capturing an exception in the file sending process by the Client terminal; the Server returns a file receiving failure message; when a Server response message is not received or the message is overtime, the embodiment of the invention sets a file transmission exception handling mechanism and a retransmission mechanism. And continuously retransmitting the failed transmission task when the retransmission time upper limit is not reached, and setting the task sending state as failure when the retransmission time upper limit is reached.

And traversing the MainTask in real time by the task management thread, matching the SubTask subtasks according to the main task found by the main file sending task in the MainTask, deleting the corresponding tasks in the MainTask and the SubTask when the states of all the matched subtasks are finished, and moving the file to a sending success directory or a sending failure directory.

(II) Server terminal

In the embodiment of the invention, the Server terminal is a Netty Server terminal. In one embodiment, the network server creates two NioEventLoopgroup thread pools, and allocates the two NioEventLoopgroup thread pools to a bossGroup (a thread) and a workGroup (a thread), wherein the bossGroup is used for processing a connection request, connection establishment and release of the Client terminal, and the workGroup thread pool is used for processing an IO request after connection establishment. The server side setting can accept a maximum of 128 connections simultaneously to meet the service requirement of high concurrent reception. The transmission Channel decoder is set to:

Lz4FrameDecoder

HeaderServerHandlerNew

wherein, the Lz4FrameDecoder decodes the received data first; the HeaderServerHandleNew realizes the receiving service, and the specific flow is as follows:

(1) And inserting a message header into the receiving task table, and creating a target file in the tmp directory.

(2) And the message header is successfully received, the file is successfully created, and a header recv succ message is sent to the Client. If the message fails, a header recv fail message is sent, and the retransmission can be supported if the message is unsuccessfully sent.

(3) And receiving the file and carrying out integrity check on the file. And if the verification is passed, moving the file to the target directory, and replying a file recv succ message to the Client. Otherwise, deleting the received residual file and deleting the queue information.

(4) And dynamically cleaning the residual queue tasks, wherein cleaning time can be set according to the size of the file.

In one embodiment, the server may set a receiving task table, where the data structure of the task table is ConcurrentHashMap < String, recvttaskinfo >, and the recvttaskinfo includes fields as shown in table 6:

numbering	Name of field	Means of
			1	taskID	Receiving task ID
2	taskName	Sending task names
			3	filename	Filename of file
4	filesize	File size
			5	destPath	Receiving a target directory
6	sameFilePolicy	Same-name file receiving strategy
			7	localip	Transmitting node IP
8	nodeName	Sending node names
			9	taskId	Sending task ID
10	taskName	Sending task names
			11	md5	MD5 check value
12	recvStatus	Receiving the result
			13	startTime	Reception start time
14	endTime	End time of reception

TABLE 6

The Client terminal and the Server terminal of the embodiment of the present invention are described in detail above. The task management module (not shown in fig. 6) between the back-end transport system and the front-end management console is described below. The module is responsible for task starting, task stopping, task restarting, task adding, modification, deletion and checking of task execution result information.

In the task list, 4 task states are set, which are a preparation state (shown as Prepare in the figure), a Processing state (shown as Processing in the figure), a Stopping state (shown as Stopping in the figure), and a Stopped state (shown as Stopped in the figure), respectively, as shown in fig. 12. For the operation of the task, the back-end transmission system and the front-end management console can interact in an RMI remote interface calling mode. And the rear-end transmission node binds and receives a task interface TaskInfRmi for the front-end management console to call, when the task is changed, the front-end management console actively calls the TaskInfRmi interface, and the task adding and deleting operation is realized in the TaskInfRmi interface.

In one embodiment, the process of adding the transmission task includes:

(1) Checking whether the task exists;

(2) If the new failure response message exists, returning the new failure response message to the front end;

(3) If not, adding the new task to the task list, and setting the task state as a Prepare state.

In one embodiment, modifying the flow of the transfer task comprises:

(1) Inquiring whether the task exists;

(2) If not, returning a modification failure response message to the front end;

(3) If so, the task list related parameters are updated.

In one embodiment, the flow of the stop task includes:

(1) Inquiring whether the task exists;

(2) Acquiring a current task state;

(3) If the task state is a preparation state (Prepare), directly setting the task state as a stop state (Stopped);

(4) If the task state is the execution state (Processing), the task state is set to be in the Stopping state (Stopping), scanning of the sending directory corresponding to the task is Stopped, and the task state is set to be in the Stopping state (Stopped) after all sending subtasks are sent and completed.

In the specific implementation process, kafka (a high-throughput distributed publish-subscribe message system) can be adopted to construct a real-time data pipeline for querying the task state, so that the transmission result record is persistent. And the transmission system writes the transmission result into the Kafka queue in real time, and the front-end management console is used as a customer subscription transmission record.

In one embodiment, the send task status message structure may be:

Topic:sendresul

Value：

srcname${separator}srcpath${separator}srcfile${separator}filesize${separator}endtime${separator}destname${separator}destpath${separator}begintime${separator}taskid${separator}taskname${separator}result${separator}scantime${separator}sendovertime

the structure of the send task status message is explained in the following table 7:

TABLE 7

The structure of receiving the task status message may be:

Topic:recvfileinfo

Value：

srcname${separator}destname${separator}destpath${separator}destfile${separator}filesize${separator}endtime${separator}taskid${separator}taskname${separator}result${separator}begintime${separator}feedback_time${separator}feedback_status

the receiving task status message structure is explained as follows in table 8:

field(s)	Name of Chinese	Type (B)
			srcname	Source node name	String
destname	Name of target node	String
			destpath	Target directory	String
destfile	Name of target file	String
			filesize	File size	String
endtime	End time of reception	String
			taskid	Task ID	String
taskname	Name of task	String
			result	Transmitting the result	int
begintime	Reception start time	String
			feedback_time	Response time	String

TABLE 8

In the aspects of availability monitoring and problem location tracking, the file transmission system also provides availability monitoring and transmission task failure alarm to a unified monitoring management platform, and an operation department can execute corresponding subsequent processing operation according to the alarm condition. Meanwhile, logs can be generated according to the operation date in the operation process of the system, and the log level can be divided into operation logs, information logs and error logs for problem analysis.

The following are different test scenarios and corresponding test effects based on the file transfer system shown in fig. 6:

(1) Test scenario 1: a single task single file, one receiving node, the test effect is shown in table 9:

file size	MD5 verification	Compression	Transmission duration	Transmission rate
					1.5G	Is that	Is that	14s	111M/s
4.5G	Whether or not	Is that	12s	372M/s

TABLE 9

(2) Test scenario 2: single task multiple files, one receiving node, the test effect is shown in table 10:

numbering

Number of transmission files

Total size of

Concurrent number

Time of flight

Transmission rate

case6

1668 are provided

50.46G

Concurrency number

1

2 minutes and 40 seconds

322M/s

case7

1668 are provided

50.46G

Number of concurrencies 3

2 minutes and 22 seconds

358M/s

case8

1668 are provided

50.46G

Concurrency number 5

2 min 13 sec

388M/s

case9

1668 are provided

50.46G

Concurrency number 10

2 minutes and 24 seconds

358M/s

TABLE 10

(3) Test scenario 3: the pressure test, the test effect is shown in table 11:

number of	Number of files transferred	Total size	Concurrent number	Time of flight	Transmission rate
						case1	23678	66.5G	Concurrency number 5	4 minutes and 46 seconds	240M/s

TABLE 11

The tests show that compared with the FTP transmission single file rate of <120M/s, the single file transmission and massive file concurrent transmission rates of the file transmission system based on the embodiment of the invention are greatly improved compared with the FTP transmission single file rate.

Based on similar inventive concepts, the embodiment of the present invention further provides a file transmission method based on an asynchronous communication framework, and preferably, the method is applicable to the file transmission device based on the asynchronous communication framework.

Fig. 13 is a flowchart of a file transfer method based on an asynchronous communication framework (e.g., netty), as shown in fig. 13, the method including:

step 131, responding to the start operation of the node, sending the node information of the node to a console through a node management module so that the console can verify the node information, and the node management module realizes the interaction between the node and the console through remote method call.

Step 132, in response to the node information verification success, receiving system information and transmission task information of the node belonging to the console through the node management module, where the system information is used for monitoring transmission operation; the system information here includes at least one of: the thread pool size of the affiliated node, the transmission thread pool size, the retransmission file threshold (i.e., the number of times), and the transmission timeout threshold (i.e., the time).

And step 133, establishing a communication connection between the node and the target node based on the asynchronous communication framework and the system information, and transmitting a corresponding file with the target node on the established communication connection according to the transmission task information.

Specifically, the communication connection between the node and the target node is established, and a connection establishment request may be initiated to the target node based on the asynchronous communication frame and the system information; and then, when the connection establishment response of the target node is received, the communication connection between the node and the target node is successfully established.

Thereafter, before transferring the file, a header may be sent to the target node over the established communication connection, the header including: file information and the node information; and responding to the successful receiving of the message header, and transmitting the file with the target node on the established communication connection according to the transmission task information.

When the target node is a plurality of nodes, the file transmission process specifically includes: firstly, acquiring a file to be transmitted in the transmission task information, and adding the file to be transmitted into a main task queue; then, adding the file to be transmitted into a subtask queue based on the target node, wherein the subtask queue is used for indicating a transmission task of the file to be transmitted and the target node, and the main task queue and the subtask queue are associated based on key values; and then, concurrently transmitting the file to be transmitted in the subtask queue to the target node.

When the file is failed to be transmitted, the file can be retransmitted based on a preset file exception handling and retransmission mechanism.

When the node is started, the node information is sent to the console through the node management module, the console verifies the node information, then when the node information is verified successfully, the node management module receives system information and transmission task information from the console, then communication connection between the node information and a target node is established based on an asynchronous communication frame and the system information, and corresponding files are transmitted with the target node on the established communication connection according to the transmission task information.

Specifically, the detailed implementation flow of each step may refer to the description in the above device embodiment, and is not described herein again.

The present embodiment also provides an electronic device, which may be a desktop computer, a tablet computer, a mobile terminal, and the like, but is not limited thereto. In this embodiment, the electronic device may be implemented by referring to the above method embodiment and the asynchronous communication frame-based file transmission apparatus/system embodiment, and the contents thereof are incorporated herein, and repeated details are not repeated.

Fig. 14 is a schematic block diagram of a system configuration of an electronic apparatus 600 according to an embodiment of the present invention. As shown in fig. 14, the electronic device 600 may include a central processor 100 and a memory 140; memory 140 is coupled to central processor 100. Notably, this figure is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications or other functions.

In one embodiment, the asynchronous communication framework based file transfer functionality may be integrated into the central processor 100. The central processor 100 may be configured to control as follows:

step 131, responding to the node starting operation, and sending the node information of the node to a console, so that the console can verify the node information;

step 132, in response to the node information being successfully verified, receiving system information and transmission task information of the node belonging to the console, where the system information is used to monitor transmission operation;

and step 133, establishing a communication connection between the node and the target node based on the asynchronous communication frame and the system information, and transmitting a corresponding file with the target node on the established communication connection according to the transmission task information.

As can be seen from the above description, in the electronic device provided in the embodiment of the present application, when a node is started, the node information is sent to the console, the console performs a verification operation on the node information, and then, when the node information is successfully verified, the system information and the transmission task information from the console are received, then, a communication connection with a target node is established based on the asynchronous communication framework and the system information, and a corresponding file is transmitted with the target node on the established communication connection according to the transmission task information.

In another embodiment, the asynchronous communication framework based file transfer device/system may be configured separately from the central processor 100, for example, the asynchronous communication framework based file transfer device/system may be configured as a chip connected to the central processor 100, and the asynchronous communication framework based file transfer function is realized by the control of the central processor.

As shown in fig. 14, the electronic device 600 may further include: communication module 110, input unit 120, audio processing unit 130, display 160, power supply 170. It is noted that the electronic device 600 does not necessarily include all of the components shown in fig. 14; furthermore, the electronic device 600 may also comprise components not shown in fig. 14, which may be referred to in the prior art.

As shown in fig. 14, the central processor 100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, the central processor 100 receiving input and controlling the operation of the various components of the electronic device 600.

The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable devices. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 100 may execute the program stored in the memory 140 to realize information storage or processing, etc.

The input unit 120 provides input to the cpu 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.

The memory 140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 140 may also be some other type of device. Memory 140 includes buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage section 142, and the application/function storage section 142 is used to store application programs and function programs or a flow for executing the operation of the electronic device 600 by the central processing unit 100.

The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging application, address book application, etc.).

The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and to receive audio input from the microphone 132 to implement general telecommunication functions. Audio processor 130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 130 is also coupled to the central processor 100, enabling recording locally through a microphone 132, and enabling locally stored sound to be played through a speaker 131.

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the file transfer method based on the asynchronous communication framework.

In summary, the embodiment of the present invention is based on the high performance asynchronous communication framework Netty, an NIO server/client transmission platform is implemented, a thread pool mode and a zero copy mechanism provided by Netty are utilized, and an Lz4 compression algorithm and a custom codec are matched to implement a high concurrency and high speed file transmission function, so that the problem of high speed and reliable transmission of large batches of files is solved, a file transmission management system with high transmission rate, safety and reliability and traceable transmission results is established, real-time file transmission can be supported, and the requirements of future banking business development are met.

The preferred embodiments of the present invention have been described above with reference to the accompanying drawings. The many features and advantages of the embodiments are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the embodiments which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the embodiments of the invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope thereof.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A file transmission method based on an asynchronous communication framework is characterized by comprising the following steps:

responding to the starting operation of the node, sending the node information of the node to a console through a node management module so that the console can conveniently verify the node information, and the node management module realizes the interaction between the node and the console through remote method calling;

responding to the successful verification of the node information, receiving system information and transmission task information of a node to which the control console belongs through the node management module, wherein the system information is used for monitoring transmission operation;

and establishing communication connection between the node and the target node based on the asynchronous communication frame and the system information, and transmitting a corresponding file with the target node on the established communication connection according to the transmission task information.

2. The method of claim 1, wherein establishing a communication connection between the node and the target node based on the asynchronous communication framework and the system information comprises:

initiating a connection establishment request to the target node based on an asynchronous communication framework and the system information;

and when the connection establishment response of the target node is received, successfully establishing the communication connection between the node and the target node.

3. The method of claim 1, wherein transmitting the corresponding file with the target node over the established communication connection according to the transmission task information comprises:

sending a message header to the target node over the established communication connection, the message header comprising: file information and the node information;

and responding to the successful receiving of the message header, and transmitting the file with the target node on the established communication connection according to the transmission task information.

4. The method of claim 3, wherein the target node is a plurality of nodes, and transmitting the corresponding file with the target node over the established communication connection according to the transmission task information comprises:

acquiring a file to be transmitted in the transmission task information, and adding the file to be transmitted into a main task queue;

adding the file to be transmitted into a subtask queue based on the target node, wherein the subtask queue is used for indicating a transmission task of the file to be transmitted and the target node, and the main task queue and the subtask queue are associated based on a key value;

and transmitting the file to be transmitted in the subtask queue to the target node.

5. The method of claim 1, further comprising:

and in response to the failure of file transmission with the target node on the established communication connection, carrying out retransmission processing on the file based on a preset file exception processing and retransmission mechanism.

6. The method according to any one of claims 1 to 5, wherein the system information comprises at least one of:

the size of the thread pool of the node, the size of the transmission thread pool, the threshold value of the retransmission file and the threshold value of the transmission overtime.

7. An apparatus for transferring a file based on an asynchronous communication framework, the apparatus comprising:

the information sending unit is used for responding to the starting operation of the node, sending the node information of the node to a console through a node management module so as to be convenient for the console to carry out verification operation on the node information, and the node management module realizes the interaction between the node and the console through remote method calling;

the information receiving unit is used for responding to the successful verification of the node information and receiving system information and transmission task information of the node to which the control console belongs through the node management module, wherein the system information is used for monitoring transmission operation;

the connection establishing unit is used for establishing communication connection between the node and the target node based on the asynchronous communication frame and the system information;

and the file transmission unit is used for transmitting the corresponding file with the target node on the established communication connection according to the transmission task information.

8. A file transfer system based on an asynchronous communication framework, the system comprising: the console, the asynchronous communication framework based file transmission device of claim 7, and the asynchronous communication framework based file receiving device, wherein the file receiving device is located at a target node, and the file receiving device and the file transmission device transmit files through an established communication connection.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any one of claims 1 to 6 when executing the program.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.