CN113347245B - File fragment breakpoint transmission method and device based on message queue - Google Patents
File fragment breakpoint transmission method and device based on message queue Download PDFInfo
- Publication number
- CN113347245B CN113347245B CN202110601917.7A CN202110601917A CN113347245B CN 113347245 B CN113347245 B CN 113347245B CN 202110601917 A CN202110601917 A CN 202110601917A CN 113347245 B CN113347245 B CN 113347245B
- Authority
- CN
- China
- Prior art keywords
- file
- task
- service bus
- sending
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/40—Bus networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
- H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the technical field of data transmission, in particular to a file fragmentation breakpoint transmission method and device based on a message queue.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a file fragment breakpoint transmission method and device based on a message queue.
Background
The microservice architecture is a new technology for deploying applications and services in the cloud. Much of the debate around microservices has focused on whether containers or other technologies perform microservices well. The microservice may run in its own program and communicate with the HTTP-type API through the lightweight device. In the service disclosure, many services can be limited by internal independent processes, and if any one of the services needs to add some function, the process scope must be reduced; in the micro-service architecture, only the required functions need to be added in a specific certain service, and the architecture of the whole process is not influenced.
Communication among services in the existing micro-service architecture is usually realized through an HTTP (hyper text transport protocol), the transmission stability of large files, the safety and the efficiency of a communication process are not guaranteed, and the transmission process cannot be quantized.
Disclosure of Invention
The embodiment of the invention provides a file fragment breakpoint transmission method and device based on a message queue, which are used for solving the technical problem of low transmission efficiency in the existing method for transmitting a large file by adopting a micro-service architecture.
In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:
a file fragment breakpoint transmission method based on a message queue comprises the following steps:
s10, a sender acquires a file to be sent and transmits information of the file to be sent to a service bus;
s20, the service bus generates a sending task according to the file size of the file to be sent and the size of a preset transmission file block, and adds the sending task into a queue to be executed of the service bus;
s30, according to a sending task existing in a queue to be executed, a supervision thread of the service bus establishes an execution thread corresponding to the sending task;
s40, starting the execution thread, acquiring an idle queue from a message queue pool of the service bus and informing a receiving party of subscribing the idle queue; the execution thread splits the sending task into N task items;
s50, the service bus sequentially informs the sender according to the N task items, the sender edits a file to be sent corresponding to the task items to obtain an edited file, transmits the edited file to the idle queue, and transmits file blocks corresponding to the edited file to the receiver one by one through the idle queue;
and S60, the receiver receives the file blocks corresponding to each edited file, if the verification code of each file block is consistent with the verification code in the corresponding task item, the sending task is completed, the service bus marks that the sending task is completed and notifies the sender of the message that the sending task is completed.
Preferably, the file fragment breakpoint transmission method based on the message queue includes: and if the sender, the service bus and/or the receiver are abnormal and can not normally transmit the file blocks, after the sender, the service bus and the receiver are recovered to be normal, continuing to complete the sending task according to the nodes transmitted by the file blocks before the abnormality.
Preferably, in step S50, before the file blocks corresponding to the edited file are transmitted to the receiving party one by one through the free queue, the edited file is encrypted;
correspondingly, in step S60, after receiving the file block corresponding to each of the edited files, the receiving side decrypts the file block.
Preferably, in step S60, after the verification code of each file block is consistent with the verification code in the corresponding task item, before the completion of the sending task, the method includes: comparing the file information in each file block with the corresponding edited file information, wherein if the file information in the file block is consistent with the corresponding edited file information, the file information of the file block is complete;
if the file information of all the file blocks received by the receiver is complete, all the file blocks are combined into a file, and the file blocks which have finished the sending task and are cached by the receiver are sent to the service bus.
Preferably, if the file information in the file block is inconsistent with the file information of the corresponding edited file, the file information of the file block is incomplete, the receiving side sends a request to the service bus to retransmit the file block, and the service bus retransmits the file block to the receiving side through the free queue until the file information of the file block received by the receiving side is complete.
Preferably, the file fragment breakpoint transmission method based on the message queue includes: and if the verification code of the file block is not consistent with the verification code in the corresponding task item, re-executing the service bus to send a sending notice corresponding to the task item to the sender, and sending the byte code of the file to be sent corresponding to the task item to the idle queue by the sender according to the sending notice.
Preferably, in step S20, if there is a task transmission in the thread of execution of the service bus, the method includes: judging whether the number of the execution threads of the current service bus reaches the maximum configuration number of the service bus or not;
if yes, the service bus supervises the operation of the execution thread;
if not, establishing an execution thread corresponding to the sending task, and adding the sending task into the execution thread.
Preferably, the information of the file to be sent includes a unique code, a file name and a total byte size of the file; each task item comprises a verification code of a file, the byte size of the file and the starting and ending positions of the bytes of the file; the file block comprises a verification code, a file name and a file byte size of the file; the edit file includes a file name and a file byte size.
Preferably, after receiving the message that the service bus marks that the sending task is completed and notifies that the sending task is completed, the sending party includes: and the sender deletes all the editing files corresponding to the task items.
The invention also provides a file fragmentation breakpoint transmission device based on the message queue, which comprises a file acquisition module, a file processing module, an execution thread establishing module, a task splitting module, a task sending module and a task checking module;
the file acquisition module is used for a sender to acquire a file to be sent and transmit the information of the file to be sent to a service bus;
the file processing module is used for generating a sending task by the service bus according to the file size of the file to be sent and the size of a preset transmission file block and adding the sending task into a queue to be executed of the service bus;
the execution thread establishing module is used for establishing an execution thread corresponding to a sending task according to the sending task existing in a queue to be executed by a supervision thread of the service bus;
the task splitting module is used for starting the execution thread, acquiring an idle queue from a message queue pool of the service bus and informing a receiver of subscribing the idle queue; the execution thread splits the sending task into N task items;
the task sending module is used for the service bus to sequentially inform the sender according to the N task items, the sender edits the file to be sent corresponding to the task items to obtain an edited file, transmits the edited file to the idle queue, and transmits file blocks corresponding to the edited file to the receiver one by one through the idle queue;
the task checking module is used for the receiver to receive the file blocks corresponding to the edited files, the sending task is completed if the verification code of each file block is consistent with the verification code in the corresponding task item, and the service bus marks that the sending task is completed and informs the sender of the message that the sending task is completed.
According to the technical scheme, the embodiment of the invention has the following advantages: the method manages the file transmission process uniformly through a sender, a service bus, a receiver and a message queue in the service bus, the service bus can receive service information of the sender and the receiver, the multi-thread of the service bus can be used for rapidly realizing the blocking transmission of sending tasks, the service bus is used for monitoring the whole file transmission process, the transmission thread of a multi-thread monitoring sending task is established to ensure the complete and efficient transmission of files, and the technical problem that the transmission efficiency is low when a micro-service architecture is used for transmitting large files in the prior art is solved.
The file fragment breakpoint transmission method based on the message queue can realize file transmission when abnormal sending or receiving, overtime retry and breakpoint continuous transmission are carried out. The breakpoint transmission method of the file fragments based on the message queue divides each sending task into a plurality of task items through the message queue of the service bus, each task item corresponds to a file block of a file to be sent for transmission, the file is encrypted and then transmitted before each transmission, so that the transmission safety between networks is achieved, each received file block is decrypted, the received file block is subjected to unified message service verification integrity, if the verification code is found not to be matched with the verification code during sending, the transmission of the file block is re-executed, and the received file is automatically transmitted to a receiving party.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.
Fig. 1 is a flowchart illustrating steps of a breakpoint transmission method for file fragments based on a message queue according to an embodiment of the present invention.
Fig. 2 is a flowchart of a breakpoint transmission method for file fragmentation based on a message queue according to an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the application provides a file fragment breakpoint transmission method and device based on a message queue, and solves the technical problem of low transmission efficiency in the existing method of adopting a micro service architecture to transmit a large file.
The first embodiment is as follows:
fig. 1 is a flowchart illustrating steps of a breakpoint transmission method for file fragmentation based on a message queue according to an embodiment of the present invention, and fig. 2 is a flowchart illustrating a breakpoint transmission method for file fragmentation based on a message queue according to an embodiment of the present invention.
As shown in fig. 1 and fig. 2, an embodiment of the present invention provides a file fragment breakpoint transmission method based on a message queue, including the following steps:
and S10, the sender acquires a file to be sent and transmits information of the file to be sent to a service bus.
It should be noted that, in step S10, the file to be transmitted that needs to be transmitted is mainly acquired, and the information of the file to be transmitted is transmitted to the service bus. In this embodiment, after obtaining a file to be sent, a sender compresses the file and caches the compressed file to the local, and then sends information of the file to be sent to a service bus micro-server, and the sender compresses the file to be sent first and then transmits the compressed file, so that the size of the file to be sent can be compressed to about 5%, the transmission efficiency of the file is improved, and the compression rate of the file can reach about 95%.
In an embodiment of the invention, the service bus comprises a sending service and a receiving service. The delivery service refers to a micro service that has a tool kit dependency of a sender, a file creation completion notification, a file delivery completion notification, and the like. Receiving a service refers to having the receiver's toolkit dependent microservice, building and implementing file handling notifications, etc. functions.
It should be noted that, on the basis of the receiving service, the receiving service may periodically send micro service information of registering itself by the receiving party to the service bus, where the micro service information includes an IP, a port, a CPU usage rate, and a remaining memory amount.
In the embodiment of the invention, the information of the file to be sent is transmitted to the service bus through the sending service sender
It should be noted that the file to be sent includes the association ID, the file name, and the file storage path, and the information of the file to be sent includes the unique code, the file name, and the total byte size of the file. The service bus receives the information of the file to be sent, firstly caches the information of the file to be sent and serializes the information to a local cache directory.
And S20, the service bus generates a sending task according to the file size of the file to be sent and the size of a preset transmission file block, and adds the sending task into a queue to be executed of the service bus.
And S30, according to the sending task existing in the queue to be executed, the supervision thread of the service bus establishes an execution thread corresponding to the sending task.
It should be noted that, in step S20 and step S30, the file to be sent is mainly generated into a sending task, and the sending task is added into a queue to be executed of the service bus, where the service bus includes a supervisory thread (the supervisory thread is for supervising the sending task to generate an execution thread) that is executed uninterruptedly, and if there is a sending task in the queue to be executed, an execution thread is generated according to the sending task. In this embodiment, the service bus divides the file size of the file to be transmitted by the size of the preset transmission file block to obtain the number of transmission file blocks, and the number of transmission file blocks forms a transmission task.
In the embodiment of the invention, the sending task generated in the service bus is placed in the static queue of the message queue pool and is stored in the database. The service bus is made up of a permanently uninterrupted thread, with a supervision mode to ensure that the service bus never stops.
S40, starting an execution thread, acquiring an idle queue from a message queue pool of a service bus and informing a receiving party of subscribing the idle queue; the execution thread will send the task split N task items. Wherein each task item comprises a verification code of the file, a byte size of the file and a starting and ending position of the byte of the file.
It should be noted that, before the execution thread executes the sending task item, a message queue corresponding to the task item needs to be established, and on the basis of the sending service and the receiving service of the service bus, the service bus informs the receiver of the established free queue information in a notification manner. The execution thread splits the sending task into N task items according to the number of the transmission file blocks, namely the number of the split task items is the same as the number of the transmission file blocks.
S50, the service bus sequentially informs a sender according to the N task items, the sender edits the file to be sent corresponding to the task items to obtain an edited file, the edited file is transmitted to an idle queue, and file blocks corresponding to the edited file are transmitted to a receiver one by one through the idle queue. And caching the edited file into a free queue and marking as a file block.
It should be noted that the receiving side receives the notification of the information for establishing the free queue sent by the service bus, and establishes a corresponding transmission relationship between the receiving side and the free queue, so that the receiving side can receive the file transmitted by the service bus. In this embodiment, after the service bus obtains that the receiving party establishes a corresponding transmission relationship with the idle queue, the service bus sends a sending notification corresponding to each task item to the sending party, the sending party sends a file to be sent corresponding to the task item according to the sending notification, the file is recorded as an edit file, the edit file is transmitted to the idle queue, the idle queue forms the edit file into a transmitted file block, and the file blocks are transmitted to the receiving party one by one through the idle queue. Wherein the file block includes a verification code, a file name, and a file byte size of the file. The edit file includes a file name and a file byte size. And after receiving the sending notification, the sender adds the byte codes appointed by the file to be sent into the appointed idle queue according to the content of the sending notification to obtain the file blocks needing to be transmitted.
In the embodiment of the invention, after the execution thread of the task is started, the transmitted task items are constructed according to the information in the sending task, each task item is a file block sending, the idle queue is obtained in the message queue pool, the notification of low memory utilization rate and high memory surplus is picked out according to the service information of receiving party receiving service registration in the service bus, the corresponding idle queue is obtained, the execution thread circularly executes transmission according to the number of the task items in the sending task, one file block corresponding to the task item is circularly transmitted every time, and then the file block is sent to the idle queue according to the sending service of the task item information notification service bus.
It should be noted that the file block contains a unique code, a subscript, and a data block. The data block is a file byte block mapped by the starting and ending positions recorded in the task item through a memory. The task item contains the subscript of the current transfer file block, the byte start position and end position of the current file block, the MD5 code (validation code) of the current file block, the status of the task item, and the like.
And S60, the receiver receives the file blocks corresponding to each edited file, if the verification code of each file block is consistent with the verification code in the corresponding task item, the sending task is completed, the service bus marks that the sending task is completed and notifies the sender of the message that the sending task is completed.
It should be noted that, when the receiving side receives a file block in the free queue, a local cache directory is established according to a unique code in the information of the file block, the file block is written into the cache directory with a subscript in the information as a file name, and the completion of the task item is notified to the service bus based on the notification of the receiving service, and the notification process carries the MD5 code of the current file block, so that the MD5 code is consistent with the MD5 code stored in the task item, which indicates that the task item is in a complete state, otherwise, the file block is retransmitted. In this embodiment, after receiving a file block according to the free queue, if the file block is cached locally at the receiving party and the service bus is notified that the receiving is successful, the notified content includes a verification code of the file block, which carries a bytecode, and the service bus compares whether the verification codes in the task item are consistent when sending, and if so, the task item is identified to complete transmission. If all task items are identified, the receiver merges all file blocks cached locally and hands over the file to be processed by the service file of the receiver, when some file blocks are found to be absent in the processing process, the service bus is informed to resend the file blocks, and the service bus is informed of the completion of the transmission of the sending task after the service processing.
In the embodiment of the invention, the execution thread can sequentially execute the transmission of the task items until the completion number of the task items is equal to the number of the transmission file blocks in the sending task, and the receiving direction sends the task completion data based on the service bus. The task completion data comprises a unique code, an association ID, a file name and the total number of file blocks.
The file fragmentation breakpoint transmission method based on the message queue manages the file transmission process uniformly through the sender, the service bus and the receiver, the service bus can receive the service information of the sender and the receiver, the transmission task blocking can be rapidly realized by utilizing multithreading of the service bus, the whole file process is supervised by data block integrity check, data blocking and combination, a multiline supervision transmission task thread is established to ensure the complete and efficient transmission of files, and the technical problem that the transmission efficiency is low when a micro-service architecture is adopted to transmit large files in the prior art is solved.
In an embodiment of the present invention, the file fragment breakpoint transmission method based on the message queue includes: and if the sender, the service bus and/or the receiver are abnormal and can not normally transmit the file blocks, after the sender, the service bus and the receiver are recovered to be normal, continuing to complete the sending task according to the nodes transmitted by the file blocks before the abnormality occurs.
It should be noted that, in the file transmission process, if a crash of the sender, the service bus and/or the receiver occurs to cause that the file cannot be normally transmitted, the state of the sending task in the service bus is not marked as completed, and after the crashed sender, service bus and/or receiver normally operate, the file transmission of the file fragmentation breakpoint transmission method based on the message queue continues to operate, so that breakpoint continuous transmission of file transmission is realized, and the problem of file transmission loss is avoided.
In one embodiment of the present invention, in step S50, before file blocks corresponding to the edited file are transmitted to the receiver one by one through the free queue, the edited file is encrypted;
correspondingly, in step S60, after receiving the file block corresponding to each edited file, the receiving side decrypts the file block.
It should be noted that, in the process of file block transmission, encryption processing is performed on the transmitted file, that is, the edited file is encrypted in the sender, and the file block received by the receiver is decrypted, so that the file block is intercepted and then decrypted in the transmission process, and the security of file transmission is ensured. In this embodiment, the key SM2 may be used for encrypting and decrypting the file.
In an embodiment of the present invention, in step S60, after the verification code of each file block is consistent with the verification code in the corresponding task item, before the completion of the sending task, the method includes: comparing the file information in each file block with the corresponding edited file information, wherein if the file information in the file block is consistent with the file information of the corresponding edited file, the file information of the file block is complete;
if the file information of all the file blocks received by the receiver is complete, all the file blocks are combined into a file, and all the file blocks which have finished sending tasks and are cached by the receiver are sent to a service bus;
if the file information in the file block is inconsistent with the file information of the corresponding edited file, the file information of the file block is incomplete, the receiving side sends a request to the service bus to transmit the file block again, and the service bus transmits the file block to the receiving side again through the idle queue until the file information of the file block received by the receiving side is complete.
It should be noted that, after receiving all the file blocks, the receiver verifies the integrity of the file blocks, so as to ensure the accuracy of file transmission. The receiver also combines all the file blocks to complete a file, and removes the combined file blocks to clear the memory of the receiver.
In one embodiment of the present invention, after the sender receives a message that the service bus marks that the sending task is completed and notifies that the sending task is completed, the method includes: and the sender deletes all the editing files corresponding to the task items.
It should be noted that the purpose of deleting all the edit files corresponding to the task items by the sender is to clear the memory of the sender. In this embodiment, the service bus receives the receiver transmission completion notification, and first notifies the sender to clear the buffer, and then identifies that the sending task is completed.
As shown in fig. 2, in an embodiment of the present invention, the method for breakpoint transmission of file fragments based on message queues includes: and if the verification code of the file block is inconsistent with the verification code in the corresponding task item, the re-execution service bus sends a sending notice corresponding to the task item to the sender, and the sender transmits the byte code of the file to be sent corresponding to the task item to the idle queue according to the sending notice.
In an embodiment of the invention, before a supervisory thread of a service bus establishes an execution thread corresponding to a sending task, the file fragmentation breakpoint transmission method based on the message queue comprises the following steps: judging whether the number of the execution threads of the current service bus reaches the maximum configuration number of the service bus;
if yes, the service bus supervises the operation of the execution thread;
and if not, establishing an execution thread corresponding to the sending task in the service bus.
It should be noted that, before the execution thread executes the send task, the service bus scans the send task queue in the message queue pool, and if no send task object exists in the message queue, the send task is added to the free queue in the message queue pool. If the message queue of the sending task exists in the message queue pool, judging whether the sending task currently executed reaches the preset maximum configuration number, if so, the service bus supervises the operation of the execution thread, namely, the service bus enters the dormancy; if the maximum configuration number does not reach the preset maximum configuration number (can be set according to the requirement), the sending task is obtained, an execution thread corresponding to the sending task is established, the sending task is added into the execution thread of the task to start the execution thread of the task, and then the execution thread of the task executes the sending of the file block. In this embodiment, the thread of execution of the task is also added to the cache after being established, and the thread of execution of a new task will not be restarted when the next service bus scans the sending task that is being executed.
The file fragment breakpoint transmission method based on the message queue can realize file transmission when abnormal sending or receiving, overtime retry and breakpoint continuous transmission are carried out. The breakpoint transmission method of the file fragments based on the message queue divides each sending task into a plurality of task items through the message queue of the service bus, each task item corresponds to a file block of a file to be sent for transmission, the file is encrypted and then transmitted before each transmission, so that the transmission safety between networks is achieved, each received file block is decrypted, the received file block is subjected to unified message service verification integrity, if the verification code is found not to be matched with the verification code during sending, the transmission of the file block is re-executed, and the received file is automatically transmitted to a receiving party.
The file fragment breakpoint transmission method based on the message queue described above is explained by using the following cases:
the method comprises the following steps that 3 existing companies are provided, wherein a company A often has a large amount of goods to be stored, the company A entrusts an express company to process the storage of the goods, the express company receives the entrusts and stores the goods of the company A in a storage company B and a storage company C, 1 salesman d is dispatched and resides in the company A, and the salesman d is mainly responsible for helping the company A to manage and send the goods; and 2 business persons e and f are dispatched to a B warehousing company and a C warehousing company respectively, and the business persons e and f entering the B warehousing company are mainly responsible for reporting warehousing surplus and receiving goods at regular time and carrying out sorting work on express delivered in batches (equivalent to the content of the step S10). Here, the job role of the employee d corresponds to the sender, the job roles of the employees e and f correspond to the receiver, and the courier company corresponds to the service bus. If company a needs to store a batch of goods (10 tons), then the batch of goods is handed to salesman d, and the salesman d informs the courier company that a batch of 10 tons of goods needs to be stored, and immediately after the notification of salesman d, the company generates a delivery task (corresponding to a delivery task) according to the size of the batch of goods and the maximum transportation volume each time and adds the delivery task to the scheduling table (corresponding to the content of step S20). The courier company has a dispatcher (corresponding to the monitoring thread), and the dispatcher has the responsibility of arranging human transportation according to the scheduling list, and the dispatcher sees that the company a needs to store goods in the scheduling list, immediately hands the transportation task to the transportation department executor g (corresponding to the execution thread) to process the service (corresponding to the content of the step S40). After receiving the delivery task, the executor g first selects a place where the goods are stored according to the storage residual amounts reported by the salesman e and f, then notifies the salesman with a large storage residual amount to wait for receiving the goods, if the storage residual amount reported by the salesman e during the transportation is large, the executor g notifies the salesman e to wait for receiving the goods in batches according to the delivery task information, the executor d sends a part of the goods to the place where the salesman e is located, and after receiving the notification, the salesman d informs the executor g of the list of the part of the goods after loading (equivalent to the content of step S50). After waiting for a part of goods to be transported, the salesman e stores the part of goods to a specified place and puts the part of goods together, and informs the express company of receiving the part of goods and informs the goods list together, and the executor g compares the list when the salesman d sends the goods, records that the goods are successfully delivered, otherwise, sends the goods again. And the executive G informs the salesperson d to send the goods list to the salesperson e after confirming that the salesperson e has received the goods. After receiving the item list, the salesperson e packages the items stored in the designated location into a whole, and in the packaging process, if the missing part of the item is found, the express company notifies the salesperson d of the missing part, and if the packaging is completed, the salesperson e delivers the item to the company B (corresponding to the content of step S60).
Example two:
the embodiment of the invention also provides a file fragmentation breakpoint transmission device based on the message queue, which comprises a file acquisition module, a file processing module, an execution thread establishing module, a task splitting module, a task sending module and a task checking module;
the file acquisition module is used for acquiring a file to be sent by a sender and transmitting the information of the file to be sent to a service bus;
the file processing module is used for generating a sending task by the service bus according to the file size of the file to be sent and the size of a preset transmission file block and adding the sending task into a queue to be executed of the service bus;
the execution thread establishing module is used for establishing an execution thread corresponding to a sending task according to the sending task existing in the queue to be executed and a supervision thread of the service bus;
the task splitting module is used for starting an execution thread, acquiring an idle queue from a message queue pool of the service bus and informing a receiving party of subscribing the idle queue; the execution thread splits the sending task into N task items;
the task sending module is used for the service bus to sequentially inform a sender according to the N task items, the sender edits the file to be sent corresponding to the task items to obtain an edited file, transmits the edited file to the idle queue, and transmits file blocks corresponding to the edited file to a receiver one by one through the idle queue;
and the task checking module is used for receiving the file blocks corresponding to each edited file by the receiver, finishing the sending task if the verification code of each file block is consistent with the verification code in the corresponding task item, marking that the sending task is finished by the service bus, and informing the sender of the message that the sending task is finished.
It should be noted that the modules in the second embodiment of the present invention are arranged corresponding to the steps in the first embodiment of the present invention, and the detailed description of the modules in the second embodiment of the present invention is omitted.
Illustratively, a computer program may be partitioned into one or more modules/units, which are stored in a memory and executed by a processor to accomplish the present application. One or more modules/units may be a series of computer program instruction segments capable of performing certain functions, the instruction segments being used to describe the execution of the computer program in the terminal device.
The terminal device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that the terminal device is not limited and may include more or fewer components than those shown, or some components may be combined, or different components, e.g., the terminal device may also include input output devices, network access devices, buses, etc.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable gate array (FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage may be an internal storage unit of the terminal device, such as a hard disk or a memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used for storing computer programs and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A file fragment breakpoint transmission method based on a message queue is characterized by comprising the following steps:
s10, a sender acquires a file to be sent and transmits information of the file to be sent to a service bus;
s20, the service bus generates a sending task according to the file size of the file to be sent and the size of a preset transmission file block, and adds the sending task into a queue to be executed of the service bus;
s30, according to a sending task existing in a queue to be executed, a supervision thread of the service bus establishes an execution thread corresponding to the sending task;
s40, starting the execution thread, acquiring an idle queue from a message queue pool of the service bus and informing a receiving party of subscribing the idle queue; the execution thread splits the sending task into N task items;
s50, the service bus sequentially informs the sender according to the N task items, the sender edits a file to be sent corresponding to the task items to obtain an edited file, transmits the edited file to the idle queue, and transmits file blocks corresponding to the edited file to the receiver one by one through the idle queue;
and S60, the receiver receives the file blocks corresponding to each edited file, if the verification code of each file block is consistent with the verification code in the corresponding task item, the sending task is completed, the service bus marks that the sending task is completed and notifies the sender of the message that the sending task is completed.
2. The message queue-based file fragment breakpoint transmission method according to claim 1, comprising: and if the sender, the service bus and/or the receiver are abnormal and can not normally transmit the file blocks, after the sender, the service bus and the receiver are recovered to be normal, continuing to complete the sending task according to the nodes transmitted by the file blocks before the abnormality.
3. The message queue-based file fragment breakpoint transmission method according to claim 1, wherein in step S50, before file blocks corresponding to the edited file are transmitted to the receiving party one by one through the free queue, the edited file is encrypted;
correspondingly, in step S60, after receiving the file block corresponding to each of the edited files, the receiving side decrypts the file block.
4. The message queue-based file fragment breakpoint transmission method according to claim 1, wherein in step S60, after the validation code of each file block is consistent with the validation code in the corresponding task item, before the sending task is completed, the method includes: comparing the file information in each file block with the corresponding edited file information, wherein if the file information in the file block is consistent with the corresponding edited file information, the file information of the file block is complete;
if the file information of all the file blocks received by the receiver is complete, all the file blocks are combined into a file, and the file blocks which have finished the sending task and are cached by the receiver are sent to the service bus.
5. The message queue-based file fragment breakpoint transmission method according to claim 4, wherein if file information in the file block is inconsistent with file information of the corresponding edited file, the file information of the file block is incomplete, the receiving side sends a request to the service bus to retransmit the file block, and the service bus retransmits the file block to the receiving side through the free queue until the file information of the file block received by the receiving side is complete.
6. The message queue-based file fragment breakpoint transmission method according to claim 1, comprising: and if the verification code of the file block is not consistent with the verification code in the corresponding task item, re-executing the service bus to send a sending notice corresponding to the task item to the sender, and sending the byte code of the file to be sent corresponding to the task item to the idle queue by the sender according to the sending notice.
7. The message queue-based file fragmentation breakpoint transmission method according to claim 1, wherein before a supervisory thread of the service bus establishes an execution thread corresponding to the sending task, the method comprises: judging whether the number of the execution threads of the current service bus reaches the maximum configuration number of the service bus or not;
if yes, the service bus supervises the operation of the execution thread;
and if not, establishing an execution thread corresponding to the sending task in the service bus.
8. The message queue-based file fragment breakpoint transmission method according to claim 1, wherein the information of the file to be transmitted includes a unique code, a file name, and a total byte size of the file; each task item comprises a verification code of a file, a byte size of the file and a starting and ending position of a byte of the file; the file block comprises a verification code, a file name and a file byte size of the file; the edit file includes a file name and a file byte size.
9. The message queue-based file fragment breakpoint transmission method according to claim 1, wherein after the sender receives the message that the service bus flag sending task is completed and notifies that the sending task is completed, the method includes: and the sender deletes all the editing files corresponding to the task items.
10. A file fragmentation breakpoint transmission device based on a message queue is characterized by comprising a file acquisition module, a file processing module, an execution thread establishing module, a task splitting module, a task sending module and a task checking module;
the file acquisition module is used for a sender to acquire a file to be sent and transmit the information of the file to be sent to a service bus;
the file processing module is used for generating a sending task by the service bus according to the file size of the file to be sent and the size of a preset transmission file block and adding the sending task into a queue to be executed of the service bus;
the execution thread establishing module is used for establishing an execution thread corresponding to a sending task according to the sending task existing in a queue to be executed by a supervision thread of the service bus;
the task splitting module is used for starting the execution thread, acquiring an idle queue from a message queue pool of the service bus and informing a receiver of subscribing the idle queue; the execution thread splits the sending task into N task items;
the task sending module is used for the service bus to sequentially inform the sender according to the N task items, the sender edits the file to be sent corresponding to the task items to obtain an edited file, transmits the edited file to the idle queue, and transmits file blocks corresponding to the edited file to the receiver one by one through the idle queue;
the task checking module is used for the receiver to receive the file blocks corresponding to the edited files, the sending task is completed if the verification code of each file block is consistent with the verification code in the corresponding task item, and the service bus marks that the sending task is completed and informs the sender of the message that the sending task is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110601917.7A CN113347245B (en) | 2021-05-31 | 2021-05-31 | File fragment breakpoint transmission method and device based on message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110601917.7A CN113347245B (en) | 2021-05-31 | 2021-05-31 | File fragment breakpoint transmission method and device based on message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113347245A CN113347245A (en) | 2021-09-03 |
| CN113347245B true CN113347245B (en) | 2022-05-20 |
Family
ID=77472914
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110601917.7A Active CN113347245B (en) | 2021-05-31 | 2021-05-31 | File fragment breakpoint transmission method and device based on message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113347245B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118710B (en) * | 2022-05-25 | 2024-04-19 | 中银金融科技有限公司 | File transmission method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106201443A (en) * | 2016-07-27 | 2016-12-07 | 福建富士通信息软件有限公司 | A kind of method and system based on the Storm how concurrent written document of streaming Computational frame |
| CN109150858A (en) * | 2018-08-01 | 2019-01-04 | 武汉索雅信息技术有限公司 | Request processing method, equipment, system and the storage medium of data interaction bus |
| CN109194731A (en) * | 2018-08-24 | 2019-01-11 | 上海电气集团股份有限公司 | A kind of concurrent real-time Data Transmission interface realizing method based on configuration software |
| CN110719228A (en) * | 2019-08-27 | 2020-01-21 | 中国船舶重工集团公司第七0九研究所 | Large data packet transmission method and device based on real-time data distribution service |
| CN112398916A (en) * | 2020-10-29 | 2021-02-23 | 北京华云安信息技术有限公司 | File transmission method and device based on HTTP (hyper text transport protocol) |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110493342B (en) * | 2019-08-21 | 2021-05-14 | 北京明朝万达科技股份有限公司 | File transmission method and device, electronic equipment and readable storage medium |
| US11178243B2 (en) * | 2019-08-23 | 2021-11-16 | Toshiba Tec Kabushiki Kaisha | Subscribe-publish network device monitoring and maintenance via a service bus |
-
2021
- 2021-05-31 CN CN202110601917.7A patent/CN113347245B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106201443A (en) * | 2016-07-27 | 2016-12-07 | 福建富士通信息软件有限公司 | A kind of method and system based on the Storm how concurrent written document of streaming Computational frame |
| CN109150858A (en) * | 2018-08-01 | 2019-01-04 | 武汉索雅信息技术有限公司 | Request processing method, equipment, system and the storage medium of data interaction bus |
| CN109194731A (en) * | 2018-08-24 | 2019-01-11 | 上海电气集团股份有限公司 | A kind of concurrent real-time Data Transmission interface realizing method based on configuration software |
| CN110719228A (en) * | 2019-08-27 | 2020-01-21 | 中国船舶重工集团公司第七0九研究所 | Large data packet transmission method and device based on real-time data distribution service |
| CN112398916A (en) * | 2020-10-29 | 2021-02-23 | 北京华云安信息技术有限公司 | File transmission method and device based on HTTP (hyper text transport protocol) |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113347245A (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110601952B (en) | Multi-channel message notification sending method and device | |
| CN112667414A (en) | Message queue-based message consumption method and device, computer equipment and medium | |
| CN112015815B (en) | Data synchronization method, device and computer readable storage medium | |
| CN111045806A (en) | Method and system for realizing delay message queue | |
| JP2006190265A (en) | Server queuing system and method thereof | |
| CN109657216B (en) | Contract generation method, device, equipment and storage medium | |
| CN111835467B (en) | Message sending method, device, computer equipment and storage medium | |
| US11848907B2 (en) | Automated email legal hold | |
| CN113347245B (en) | File fragment breakpoint transmission method and device based on message queue | |
| CN111917630B (en) | Data transmission method, data transmission device, storage medium and electronic device | |
| CN114598749A (en) | Service access method and device | |
| CN114637611A (en) | Information processing method and device based on message queue and computer equipment | |
| US9244918B2 (en) | Locating electronic documents | |
| CN115525449B (en) | Micro-service data transmission system, method and storage medium | |
| CN116308236A (en) | Mail processing method, mail processing device, electronic equipment and storage medium | |
| CN115034717A (en) | Method, device, equipment and computer readable medium for monitoring logistics menu | |
| CN103678614A (en) | Information processing method for unlocking industry security administration | |
| CN107704557B (en) | Processing method and device for operating mutually exclusive data, computer equipment and storage medium | |
| US10749632B2 (en) | Smart integrated cyclic data transport | |
| CN102880825A (en) | Method and system for efficiently calling hardware encryption equipment in UNIX/LINUX environment | |
| CN111581244B (en) | Method, system and equipment for orderly synchronizing service transaction data of heterogeneous system | |
| CN113179333B (en) | File transmission and real-time computing system and method for block chain | |
| CN115564588A (en) | Transaction data processing method and device based on message queue and server | |
| JP2023019934A (en) | E-mail processing apparatus and e-mail processing program | |
| CN115630120A (en) | Data synchronization method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |