CN112035407B - File transmission system and transmission method - Google Patents
File transmission system and transmission method Download PDFInfo
- Publication number
- CN112035407B CN112035407B CN202010904595.9A CN202010904595A CN112035407B CN 112035407 B CN112035407 B CN 112035407B CN 202010904595 A CN202010904595 A CN 202010904595A CN 112035407 B CN112035407 B CN 112035407B
- Authority
- CN
- China
- Prior art keywords
- file
- transmitted
- sub
- files
- threads
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 107
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000012546 transfer Methods 0.000 claims description 35
- 230000008569 process Effects 0.000 description 7
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 description 1
- YTAHJIFKAKIKAV-XNMGPUDCSA-N [(1R)-3-morpholin-4-yl-1-phenylpropyl] N-[(3S)-2-oxo-5-phenyl-1,3-dihydro-1,4-benzodiazepin-3-yl]carbamate Chemical compound O=C1[C@H](N=C(C2=C(N1)C=CC=C2)C1=CC=CC=C1)NC(O[C@H](CCN1CCOCC1)C1=CC=CC=C1)=O YTAHJIFKAKIKAV-XNMGPUDCSA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a file transmission system and a file transmission method, wherein the system comprises a file scanning module, a transmission control module, a thread pool module and a file transmission module; the file scanning module is used for scanning all files to be transmitted in a designated storage path according to timer control, generating a file linked list to be transmitted, and storing relevant information of all files to be transmitted in the file linked list to be transmitted; the transmission control module is used for dividing a file linked list to be transmitted into a plurality of sub-linked lists according to the number of the configured sub-threads, and each sub-linked list stores the related information of a plurality of files to be transmitted; the thread pool module is used for generating a corresponding number of sub threads according to the configuration parameters; the file transmission module comprises a plurality of sub-threads, wherein each sub-thread is used for transmitting a plurality of corresponding files to be transmitted to the network node storage equipment. When the FTP protocol is used for transmitting the files, the invention can realize real-time, large-batch, efficient and reliable file transmission.
Description
Technical Field
The present invention relates to the field of file transfer technologies, and in particular, to a file transfer system and a file transfer method.
Background
Currently, with the acceleration of the 5G communications era, massive storage data files are generated on a network server, and in order to release server storage space and store the data files, the massive data files need to be dumped to a network storage node device.
The common solutions mostly adopt universal FTP software or manual download server file processing by executing WINDOWS FTP command script mode. The method has the problems of poor instantaneity, low efficiency, incapability of continuous transmission, incapability of simultaneous transmission in large batches and the like.
Therefore, how to transfer these files to the network storage node device in real time, in large quantities, efficiently and reliably becomes a significant problem that needs to be solved by the industry.
Disclosure of Invention
The embodiment of the invention provides a file transmission system and a file transmission method, which are used for solving the defects that the existing network server file has poor real-time performance, low efficiency, incapability of continuous transmission and incapability of large-batch simultaneous transmission when in FTP transmission, and realizing real-time, large-batch, efficient and reliable file transmission.
According to an aspect of an embodiment of the present invention, there is provided a file transfer system, including a file scanning module, a transfer control module, a thread pool module, and a file transfer module;
the file scanning module is used for scanning all files to be transmitted in a specified storage path according to timer control, generating a file linked list to be transmitted, and storing relevant information of all files to be transmitted in the file linked list to be transmitted;
the transmission control module is used for dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of configured sub-threads, and each sub-linked list stores the related information of a plurality of files to be transmitted;
the thread pool module is used for generating a corresponding number of sub threads according to the configuration parameters;
the file transmission module comprises a plurality of sub-threads, wherein each sub-thread is used for transmitting a plurality of corresponding files to be transmitted to the network node storage equipment.
Based on the technical scheme, the invention can also make the following improvement in real time.
Optionally, the file list to be transmitted includes a file name, a file storage path, a continuous transmission state and transmitted bytes of each file to be transmitted.
Optionally, the number of the sub-linked lists is equal to the number of the sub-threads.
Optionally, the file transfer module is configured to divide the file linked list to be transferred into a plurality of sub-linked lists according to the number of configured sub-threads, where the dividing includes:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
Optionally, each of the sub-threads is configured to transmit a corresponding plurality of files to be transmitted to the network node storage device:
and for any one of the sub-threads, reading each file to be transmitted according to the file name and the file storage path of each file to be transmitted corresponding to the any one of the sub-threads, and transmitting each file to be transmitted to the network node storage device.
Optionally, the transmitting each file to be transmitted to the network node storage device includes:
and for any file to be transmitted of the sub-thread, if the continuous transmission state of any file to be transmitted is yes, the remaining bytes of any file to be transmitted are continuously transmitted to the network node storage device according to the transmitted bytes.
Optionally, the file transfer module further includes a counter;
the counter is used for accumulating and counting the finished sub-threads, wherein when the sub-threads completely transmit the corresponding files to be transmitted to the network node storage equipment, the sub-threads are finished;
the file transmission module is further configured to send a file transmission task completion signal to the transmission control module when the accumulated count of the counter is equal to the total number of sub-threads.
According to a second aspect of an embodiment of the present invention, there is provided a file transfer method, including:
scanning all files to be transmitted under a specified storage path according to timer control, and generating a file linked list to be transmitted, wherein the file linked list to be transmitted stores relevant information of all files to be transmitted;
dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of the configured sub-threads, wherein each sub-linked list stores the related information of a plurality of files to be transmitted;
and transmitting a plurality of files to be transmitted corresponding to each sub-linked list to the network node storage equipment based on each sub-thread, wherein each sub-thread corresponds to each sub-linked list.
Optionally, the number of the files to be transmitted corresponding to the sub-linked list is calculated by the following method:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
Optionally, the method further comprises:
accumulating and counting the finished sub-threads, wherein when the sub-threads completely transmit the corresponding files to be transmitted to the network node storage equipment, the sub-threads are finished;
and when the accumulated count is equal to the total number of the sub-threads, sending a file transmission task completion signal to the transmission control module.
The file transmission system and the file transmission method provided by the embodiment of the invention can realize real-time, large-scale, efficient and reliable file transmission when the file is transmitted by using the FTP protocol.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a file transfer system according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a file transfer method according to an embodiment of the present invention;
FIG. 3 is a schematic overall flow chart of a file transfer method according to an embodiment of the present invention;
FIG. 4 is a schematic workflow diagram of a document scanning module;
FIG. 5 is a schematic diagram of a workflow of a transmission control module;
FIG. 6 is a schematic workflow diagram of a thread pool module;
fig. 7 is a schematic workflow diagram of a file transfer module.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a file transfer system is provided, which includes a file scanning module, a transfer control module, a thread pool module, and a file transfer module. The file scanning module is used for scanning all files to be transmitted in a specified storage path according to timer control, generating a file linked list to be transmitted, and storing relevant information of all files to be transmitted in the file linked list to be transmitted; the transmission control module is used for dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of the configured sub-threads, and each sub-linked list stores the related information of a plurality of files to be transmitted; the thread pool module is used for generating a corresponding number of sub threads according to the configuration parameters; the file transmission module comprises a plurality of sub-threads, wherein each sub-thread is used for transmitting a plurality of corresponding files to be transmitted to the network node storage equipment.
It can be appreciated that based on the drawbacks of the solutions in the background art, the embodiment of the present invention provides a file transfer system, which is suitable for application software and communication systems developed by all operating system platforms, including, but not limited to, WINDOWS, LINUX, which is developed under the above-mentioned operating systems, and mainly includes a file scanning module, a transfer control module, a thread pool module, and a file transfer module. The file scanning module is mainly responsible for scanning all files to be transmitted and generating a file linked list to be transmitted, and the transmission control module divides the file linked list to be transmitted into a plurality of sub-linked lists according to the number of the sub-threads, wherein one sub-linked list corresponds to one sub-thread. And the thread pool module generates a corresponding number of sub threads according to the configuration parameters. The file transmission module comprises a plurality of sub-threads, and each sub-thread is responsible for transmitting a plurality of files to be transmitted expressed in a corresponding sub-linked list to the network node storage device.
When the embodiment of the invention uses the FTP protocol to transmit the files, the files can be transmitted in real time, in large batch, efficiently and reliably.
As a possible implementation manner, the file name, the file storage path, the continuous transmission state and the transmitted bytes of each file to be transmitted are included in the file to be transmitted linked list.
It can be understood that when the file scanning module scans each file to be transmitted in the designated storage path, first, it is determined whether the file to be transmitted is in a writing state, if the file to be transmitted is in the writing state, the relevant information of the file to be transmitted is not written into the file linked list to be transmitted, and if the file to be transmitted is not in the writing state, the relevant information of the file to be transmitted is written into the file linked list to be transmitted, and the files added into the file linked list to be transmitted are all files to be transmitted.
The structure of the file list to be transmitted is shown in table 1, and includes parameters such as fileName, filePath, resumeState, and transferred size. Wherein the filePath is a file path of a file to be transmitted; the default state of resumeState is false, that is, the file to be transmitted is not in a continuous transmission state, and the file is directly transmitted, when the state is true, the file is continuously transmitted, the transferring size is the size of the file which is transmitted, and when the file needs to be continuously transmitted, the system uses the parameter and transmits the file from the file position.
TABLE 1
As a possible implementation manner, the file transmission module, configured to divide the file linked list to be transmitted into a plurality of sub-linked lists according to the number of configured sub-threads, includes:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
It can be understood that the transmission control module divides the to-be-transmitted linked list into a plurality of sub-linked lists according to the number of the sub-threads, wherein the number of the sub-linked lists is equal to the number of the sub-threads, wherein the number of to-be-transmitted files represented in each sub-linked list is equal, and if a plurality of to-be-transmitted files cannot be evenly distributed in each sub-linked list at last, the to-be-transmitted files are equally distributed in a plurality of sub-linked lists at the front. For example, there are 94 files to be transmitted and 10 sub-threads, so that each sub-linked list is firstly evenly distributed with 9 files to be transmitted, and the other 4 files to be transmitted are evenly distributed into 4 sub-linked lists therein.
The information stored in each sub-linked list is similar to the information stored in the file linked list to be transmitted, and the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
As a possible implementation manner, each sub-thread is configured to transmit a corresponding plurality of files to be transmitted to the network node storage device: and for any one of the sub-threads, reading each file to be transmitted according to the file name and the file storage path of each file to be transmitted corresponding to the any one of the sub-threads, and transmitting each file to be transmitted to the network node storage device.
It can be understood that the thread pool module generates a corresponding number of sub-threads according to the configuration parameters, each sub-thread corresponds to each sub-linked list, and each sub-thread transmits a plurality of files to be transmitted, which are represented in the corresponding sub-linked list, to the network node storage device.
As a possible implementation manner, transmitting each file to be transmitted to the network node storage device includes:
and for any file to be transmitted of the sub-thread, if the continuous transmission state of any file to be transmitted is yes, the residual bytes of any file to be transmitted are continuously transmitted to the network node storage device according to the transmitted bytes.
It can be understood that if the continuous transmission state of a certain file to be transmitted is false, the file to be transmitted is directly transmitted to the network node storage device; if the continuous transmission state of a certain file to be transmitted is wire, transmitting the rest part of the file to be transmitted to the network node storage equipment according to the transmitted bytes of the file to be transmitted, and completing the breakpoint continuous transmission of the file to be transmitted.
As one possible real-time approach, the file transfer module further comprises a counter;
the counter is used for accumulating and counting the finished sub-threads, wherein when the sub-threads completely transmit the corresponding files to be transmitted to the network node storage equipment, the sub-threads are finished;
and the file transmission module is also used for sending a file transmission task completion signal to the transmission control module when the accumulated count of the counter is equal to the total number of the sub-threads.
It can be understood that the file transmission module further includes a counter, wherein after one sub-thread transmits all the corresponding files to be transmitted to the network node storage device, the count of the counter is accumulated by 1, and when the accumulated count of the counter is equal to the total number of the sub-threads, that is, all the sub-threads of the round are transmitted, the file transmission module sends a file transmission task completion signal to the transmission control module.
Referring to fig. 2, there is provided a file transfer method including: s1, scanning all files to be transmitted under a specified storage path according to timer control, and generating a file linked list to be transmitted, wherein the file linked list to be transmitted stores relevant information of all files to be transmitted; s2, dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of configured sub-threads, wherein each sub-linked list stores the related information of a plurality of files to be transmitted; and S3, each sub-thread is used for transmitting a plurality of files to be transmitted corresponding to each sub-linked list to the network node storage equipment, wherein each sub-thread corresponds to each sub-linked list.
The number of the files to be transmitted corresponding to the sub-linked list is calculated in the following manner:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
The method comprises the steps that in the process that a sub-thread transmits a file to be transmitted, the finished sub-thread is accumulated and counted, and when the sub-thread transmits the corresponding file to be transmitted to a network node storage device, the sub-thread is finished; and when the accumulated count is equal to the total number of the sub-threads, sending a file transmission task completion signal to the transmission control module.
The main flow of the file transmission method provided by the embodiment of the invention can be seen in fig. 3, which includes:
a. starting (100);
b. initializing a configuration parameter transmission START signal (101);
c. a timer control (102);
d. intelligent scanning (103) of files to be transmitted;
c. sending a STOP signal to a timer control (104);
d. transmitting an operating signal to a transmission control module (105);
e. transmitting a task state control (106);
f. intelligent processing (107) of a file linked list to be transmitted;
g. sending a work signal to the thread pool module (108);
h. an operating state control (109);
g. n sub-threads (110) are generated that call the execution file transfer function.
The file scanning module, see fig. 4, has the following working procedures:
a. receiving a START/STOP control signal (200);
b. the START control signal decision (201) is that step d is performed if the START signal is the START signal, otherwise step c is performed.
c. Stopping the timer (202);
d. starting a timer (203), starting the timer, and checking whether step e can be performed according to the system time polling.
e. And (3) judging the working time (204), and executing the step f if the working time is judged to be true.
f. And (5) judging the completion of file writing (205), scanning all files in the current path, and establishing a file linked list. Judging whether the file to be transmitted is occupied by other processes, if the file is still in a writing operation, marking the file as false, and indicating that the file cannot be transmitted; otherwise marked true, the representation may be transmitted.
g. Generating a writing completion file transmission chain table (206), and generating a file transmission chain table to be transmitted according to the node structure shown in the table I by using all files which can be transmitted according to the file chain table in the step f.
h. And (3) sending the STOP signal to a timer control (207), sending the STOP signal to a file scanning module, stopping the timer after the file scanning module receives the processing, and not executing the working step triggered by starting the timer until a new START signal is received.
i. And sending a working signal to the transmission control module, informing the transmission control module to start working, and transmitting the file transmission linked list to be tested and the configuration data as parameters to the transmission control module.
The operation flow of the transmission control module, see fig. 5, mainly comprises:
a. and the receiving work control signal (301) is used for receiving the work signal sent by the file scanning module and the transmission task completion signal sent by the file transmission module.
b. Judging the task (302), judging the type of the task to be processed, and executing the step c if the type of the task is a processing linked list; and if the transmission task is completed, executing the step d.
c. And calculating (303), namely calculating a remainder Y=size mode N and a file number count= (SIZE-Y)/N which is distributed evenly by each thread according to the set thread number N of the configuration data and the number SIZE of the files to be transmitted in the file linked list.
d. And equally dividing the linked list (304), and generating a sub-linked list array fileList [ n ], namely n sub-linked lists of the type shown in table one according to the calculation result of the step 303. The SIZE NODEs NODE of the file chain table are averagely inserted into n sub-chain tables, and each word chain table has Count NODEs NODE.
e. Y=0, and the remaining nodes, if Y is not equal to 0, step f is performed (305).
f. The remaining nodes process (306) processes the remaining Y nodes of the file linked list. Since Y < n, these Y nodes are inserted into Y sub-linked lists of file [0], file [1], file [ Y-1], etc., respectively.
g. And sending a working signal to a thread pool module (307), and sending a starting working signal to a file transmission module, wherein signal parameters comprise a sub-linked list and configuration information.
The workflow of the file transfer module can be seen in fig. 6, including:
a. and receiving a working control signal (400), and receiving a working signal from the transmission control module and a thread ending signal sent by the sub-thread.
b. Task judgment (401), branch judgment, if the task judgment is the working signal of the transmission control module, executing step g; if the thread end signal is present, step c is performed.
c. The counter is incremented by 1 (402) and the counter value Num is incremented once every time a child thread is finished.
d. Num > =n judgment (403), judges the magnitudes of the number of threads Num ended and the total number of threads n. And E, if the thread is completely finished and the result is true, executing the step e.
e. And sending a task completion signal (404) to the transmission control module.
f. Ending (405), the counter value Num, num=0 is cleared. Waiting for the next round of work.
g. Generating a sub-thread (406), generating the sub-thread according to the signal parameters, calling a file transmission module, starting to execute a file transmission function, generating n sub-threads altogether, and sending a sub-thread ending signal to a thread pool module after the sub-thread is ended.
The workflow of the file transfer module, see fig. 7, includes:
a. starting (500);
b. a timer (501) is started, connection timer parameters are set according to parameters such as IP (Internet protocol), port number, user name, user password and the like of a server and the like contained in configuration data, and the timer is started.
c. And (4) checking the trigger time (502), checking the trigger time of the timer, and executing the step d if the trigger time is up.
d. Connection status OK check (503) checks whether the communication connection with the server is OK. If true, executing the step f; if false, initiate the connection request to the server. Step e is then performed.
e. A success OK check (504) is connected, the response returned by the server is checked, if true, the step f is executed; if false, then step c is performed.
f. Stopping the timer (505);
g. and transmitting a transmission signal (506), and transmitting the transmission signal to inform the module to process the file nodes in the transmission chain table.
h. And receiving a signal (507), and receiving a processing linked list signal sent by the module.
i. Checking an empty linked list (508), checking whether the transmission linked list is empty, if true, judging that the task of the module 4 is finished by the system, and executing the step j; otherwise, executing the steps.
j. The sub-thread end signal is sent (509), and the current module transmits the task end to send the sub-thread end signal to module 3.
k. Ending (510), the current module transfers that the task has completed and the thread lifetime ends.
The file transmission module transmits the file to be transmitted in the following process:
l, head taking node transmitting file (511): and taking the head node from the linked list, reading file data according to the file name and the file path contained in the node according to the node continuous transmission state resume information, and transmitting the file to the server. If the resumeState is false, reading data from the file header and transmitting the file; if the resumeState is true, the transferedSize value is read and the file is transferred from the file location.
m, transmission completion check (512): it is checked whether the file transfer task is completed. If true, executing the step n; otherwise, executing the step q.
n, delete file (513): the system performs automatic deletion of the completed file in the path according to the header file path information.
o, delete head node (514): and completing the transmission task of the head node file, and deleting the head node.
p, transmit transmission signal (515): and sending a transmission signal, and informing the module to process the file nodes in the transmission chain table.
q, modify header node (516): as shown in table 1, the transfersize parameter value in the header node is modified and the transferred file size is recorded.
r, network link failure check (517): checking whether the network is disconnected, if true, executing step s.
s, modify head node (518): and modifying the head continuous transmission state of the head node, and modifying the resumeState to true. Step b is then performed.
The embodiment of the invention provides a file transmission system and a file transmission method, which can circularly transmit a large number of files, and can realize real-time, large-batch, efficient and reliable file transmission when the file is transmitted by using an FTP protocol.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (7)
1. The file transmission system is characterized by comprising a file scanning module, a transmission control module, a thread pool module and a file transmission module;
the file scanning module is used for scanning all files to be transmitted in a specified storage path according to timer control, generating a file linked list to be transmitted, and storing relevant information of all files to be transmitted in the file linked list to be transmitted;
the transmission control module is used for dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of configured sub-threads, and each sub-linked list stores the related information of a plurality of files to be transmitted;
the thread pool module is used for generating a corresponding number of sub threads according to the configuration parameters;
the file transmission module comprises a plurality of sub-threads, wherein each sub-thread is used for transmitting a plurality of corresponding files to be transmitted to the network node storage equipment;
the file list to be transmitted comprises a file name, a file storage path, a continuous transmission state and transmitted bytes of each file to be transmitted;
the file transmission module is configured to divide the file linked list to be transmitted into a plurality of sub-linked lists according to the number of configured sub-threads, and includes:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
2. The file transfer system of claim 1, wherein the number of sub-linked lists is equal to the number of sub-threads.
3. The file transfer system of claim 2, wherein each of the sub-threads for transferring a corresponding plurality of files to be transferred to the network node storage device comprises:
and for any one of the sub-threads, reading each file to be transmitted according to the file name and the file storage path of each file to be transmitted corresponding to the any one of the sub-threads, and transmitting each file to be transmitted to the network node storage device.
4. A file transfer system in accordance with claim 3 wherein said transferring each file to be transferred to a network node storage device comprises:
and for any file to be transmitted of the sub-thread, if the continuous transmission state of any file to be transmitted is yes, the remaining bytes of any file to be transmitted are continuously transmitted to the network node storage device according to the transmitted bytes.
5. The file transfer system of claim 1, wherein the file transfer module further comprises a counter;
the counter is used for accumulating and counting the finished sub-threads, wherein when the sub-threads completely transmit the corresponding files to be transmitted to the network node storage equipment, the sub-threads are finished;
the file transmission module is further configured to send a file transmission task completion signal to the transmission control module when the accumulated count of the counter is equal to the total number of sub-threads.
6. A file transfer method of a file transfer system according to any one of claims 1 to 5, comprising:
scanning all files to be transmitted under a specified storage path according to timer control, and generating a file linked list to be transmitted, wherein the file linked list to be transmitted stores relevant information of all files to be transmitted;
dividing the file linked list to be transmitted into a plurality of sub-linked lists according to the number of the configured sub-threads, wherein each sub-linked list stores the related information of a plurality of files to be transmitted;
transmitting a plurality of files to be transmitted corresponding to each sub-linked list to network node storage equipment based on each sub-thread, wherein each sub-thread corresponds to each sub-linked list;
the number of the corresponding files to be transmitted in the sub-linked list is calculated by the following method:
calculating a remainder Y=size mode n, wherein SIZE is the number of all files to be transmitted corresponding to a file linked list to be transmitted, and n is the number of sub-threads;
if y=0, the number of files to be transmitted allocated equally by each sub-thread count= (SIZE-Y)/n;
if Y is not equal to 0, Y files to be transmitted are evenly distributed to the first Y sub-linked lists;
the file name, the file storage path, the continuous transmission state and the transmitted bytes of each corresponding file to be transmitted are stored in each sub-linked list.
7. The file transfer method according to claim 6, further comprising:
accumulating and counting the finished sub-threads, wherein when the sub-threads completely transmit the corresponding files to be transmitted to the network node storage equipment, the sub-threads are finished;
and when the accumulated count is equal to the total number of the sub-threads, sending a file transmission task completion signal to the transmission control module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010904595.9A CN112035407B (en) | 2020-09-01 | 2020-09-01 | File transmission system and transmission method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010904595.9A CN112035407B (en) | 2020-09-01 | 2020-09-01 | File transmission system and transmission method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112035407A CN112035407A (en) | 2020-12-04 |
| CN112035407B true CN112035407B (en) | 2023-10-31 |
Family
ID=73590848
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010904595.9A Active CN112035407B (en) | 2020-09-01 | 2020-09-01 | File transmission system and transmission method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035407B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115208877B (en) * | 2022-07-21 | 2024-02-27 | 中国建设银行股份有限公司 | File transmission method, device, equipment, medium and product |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103237049A (en) * | 2013-03-18 | 2013-08-07 | 北京易华录信息技术股份有限公司 | File transfer method and device |
| WO2016165482A1 (en) * | 2015-08-07 | 2016-10-20 | 中兴通讯股份有限公司 | File transfer method and apparatus |
| CN109286677A (en) * | 2018-10-25 | 2019-01-29 | 北京北信源信息安全技术有限公司 | A kind of method and device of the file transmission based on FTP |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7756938B2 (en) * | 2007-10-03 | 2010-07-13 | International Business Machines Corporation | Eliminating redundancy of attachments in email responses |
| CN104348901A (en) * | 2014-06-30 | 2015-02-11 | 腾讯科技（深圳）有限公司 | File transmission method, file transmission device and file transmission system |
| US10545915B2 (en) * | 2015-02-02 | 2020-01-28 | Quantum Corporation | Recursive multi-threaded file system scanner for serializing file system metadata exoskeleton |
| KR102065958B1 (en) * | 2017-11-13 | 2020-02-11 | 유한회사 이노릭스 | Method and system for managing data transfer |
-
2020
- 2020-09-01 CN CN202010904595.9A patent/CN112035407B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103237049A (en) * | 2013-03-18 | 2013-08-07 | 北京易华录信息技术股份有限公司 | File transfer method and device |
| WO2016165482A1 (en) * | 2015-08-07 | 2016-10-20 | 中兴通讯股份有限公司 | File transfer method and apparatus |
| CN109286677A (en) * | 2018-10-25 | 2019-01-29 | 北京北信源信息安全技术有限公司 | A kind of method and device of the file transmission based on FTP |
Non-Patent Citations (1)
| Title |
|---|
| 基于FTP协议的文件传输服务器的研究;赵强;《信息科技》(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112035407A (en) | 2020-12-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106302632B (en) | Downloading method of basic mirror image and management node | |
| CN112035407B (en) | File transmission system and transmission method | |
| CN112887196A (en) | Message sending method, system, device, equipment and readable storage medium | |
| CN109697112B (en) | Distributed intensive one-stop operating system and implementation method | |
| CN108063809B (en) | Machine equipment data acquisition method and acquisition system | |
| CN106535101B (en) | Bluetooth communication method, system and wireless router | |
| CN112969198A (en) | Data transmission method, terminal and storage medium | |
| CN115412609B (en) | Service processing method, device, server and storage medium | |
| CN108830724B (en) | Resource data packet processing method and terminal equipment | |
| CN112865990A (en) | File upgrading method, system and device | |
| KR100662167B1 (en) | Method and system for transmitting a chain of messages for database | |
| CN115766696A (en) | File uploading method and device, electronic equipment and storage medium | |
| CN115705259A (en) | Fault processing method, related device and storage medium | |
| CN112055058A (en) | Data storage method and device and computer readable storage medium | |
| JPH0991219A (en) | Fault monitoring system | |
| CN113672248A (en) | Patch acquisition method, device, server and storage medium | |
| CN115580610B (en) | Batch file uploading method, device, equipment and readable storage medium | |
| CN108874564A (en) | A kind of inter-process communication methods, electronic equipment and readable storage medium storing program for executing | |
| CN109325134A (en) | A kind of thumbnail management method, thumbnail managing device and mobile terminal | |
| CN114578783B (en) | Remote debugging method and system for industrial field device | |
| CN113127220B (en) | Data issuing method and device and electronic equipment | |
| JP3937371B2 (en) | Competition control method and competition control system | |
| CN116347043A (en) | Method for sending server-side asynchronous IO, terminal and storage medium | |
| CN109361586B (en) | CANopen slave station starting method and overall planning manager thereof | |
| CN113992732B (en) | Terminal management control method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |