CN112559461A - File transmission method and device, storage medium and electronic equipment - Google Patents
File transmission method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112559461A CN112559461A CN202011570095.2A CN202011570095A CN112559461A CN 112559461 A CN112559461 A CN 112559461A CN 202011570095 A CN202011570095 A CN 202011570095A CN 112559461 A CN112559461 A CN 112559461A
- Authority
- CN
- China
- Prior art keywords
- task
- transmission
- message
- target file
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a file transmission method and device, a storage medium and electronic equipment, wherein when a transmission request of a target file sent by a service system is received, a transmission task of the target file can be created based on transmission information corresponding to the transmission request, a task message of the transmission task is transmitted to a transmission service node through a message queue so as to trigger the transmission service node to execute the transmission task, and after the execution process of the transmission task is completed, an execution result is fed back to the message queue. And when the execution result of the transmission task is received in the message queue, the execution result can be obtained from the message queue, and the task state of the transmission task is updated according to the execution result so as to finish the transmission process of the target file. By applying the method provided by the invention, the business system can call the preset interface to send the transmission request, and the trigger service completes the file transmission process without executing the file transmission operation by self, thereby reducing the resource consumption of the business system and increasing the business processing capacity of the business system.
Description
Technical Field
The present invention relates to the field of data transmission, and in particular, to a file transmission method and apparatus, a storage medium, and an electronic device.
Background
With the continuous development of business of enterprise organizations, in order to meet business requirements, a plurality of business systems are generally deployed to implement corresponding business functions.
Data generated by each business system in the business processing process is usually stored in the form of files in the file storage system corresponding to each business system. Due to the processing requirements of each business system, file interaction is often required between each business system.
At present, when a certain service system transmits a file to another service system, the service system usually connects to a file storage system of another service system to complete the transmission of the file. In an actual transmission scenario, each service system generally corresponds to different types of file storage systems, when a service system needs to transmit files to different service systems, different transmission operation flows need to be adopted for the different types of file storage systems, the transmission process is complex, so that the transmission efficiency of files between the service systems is low, and the system load is large.
Disclosure of Invention
In view of this, embodiments of the present invention provide a file transmission method to solve the problem that the transmission efficiency of files between service systems is low, so that the system load is large.
The embodiment of the invention also provides a file transmission device which is used for ensuring the actual realization and application of the method.
In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:
a file transfer method, comprising:
when a transmission request of a target file sent by a service system is received, determining transmission information corresponding to the target file;
creating a transmission task corresponding to the target file based on the transmission information, and initializing a task state of the transmission task;
determining a task message corresponding to the transmission task, and sending the task message to a pre-established message queue, so that a transmission service node which monitors the message queue in real time executes the transmission task after monitoring the task message, determines an execution result of the transmission task after completing an execution process of the transmission task, and sends the execution result of the transmission task to the message queue;
and when the message queue receives the execution result of the transmission task, acquiring the execution result of the transmission task from the message queue, updating the task state of the transmission task according to the execution result of the transmission task, and finishing the transmission process of the target file.
Optionally, the method, after sending the task message to the message queue, further includes:
and when the task message is not successfully received by the message queue, sending the task message to the message queue again.
Optionally, in the method, after monitoring the task message, the transmission service node executes a process of transmitting the task, including:
after the transmission service node monitors the task message, acquiring the task message from the message queue, and verifying the validity of the task message;
if the task message passes validity verification, determining a first storage system and a second storage system corresponding to the transmission task, wherein the first storage system is a source storage system for storing the target file, and the second storage system is a to-be-transmitted storage system corresponding to the target file;
judging whether the target file exists in the second storage system or not;
and if the target file does not exist in the second storage system, reading the target file from the first storage system, and writing the read target file into the second storage system.
The above method, optionally, further includes:
and if the target file exists in the second storage system, ending the execution process of the transmission task.
Optionally, in the method, after the task message is sent to the message queue, the task state of the transmission task is updated to be executing; the process of updating the task state of the transmission task according to the execution result of the transmission task includes:
determining the current task state of the transmission task, and judging whether the current task state of the transmission task is being executed;
if the current task state of the transmission task is in execution, updating the task state of the transmission task to be successful in execution or failed in execution according to the execution result of the transmission task;
and if the current task state of the transmission task is not the executing state, recording abnormal information.
The above method, optionally, further includes:
when receiving a transmission query request of the target file sent by the service system, determining the current task state of the transmission task, generating a corresponding query result according to the current task state of the transmission task, and sending the query result to the service system.
A file transfer device comprising:
the system comprises a first determining unit, a second determining unit and a third determining unit, wherein the first determining unit is used for determining transmission information corresponding to a target file when a transmission request of the target file sent by a service system is received;
the creating unit is used for creating a transmission task corresponding to the target file based on the transmission information and initializing the task state of the transmission task;
a second determining unit, configured to determine a task message corresponding to the transmission task, and send the task message to a pre-established message queue, so that a transmission service node that monitors the message queue in real time executes the transmission task after monitoring the task message, and after completing an execution process of the transmission task, determines an execution result of the transmission task, and sends the execution result of the transmission task to the message queue;
and the updating unit is used for acquiring the execution result of the transmission task from the message queue when the message queue receives the execution result of the transmission task, updating the task state of the transmission task according to the execution result of the transmission task, and ending the transmission process of the target file.
The above apparatus, optionally, further comprises:
and the sending unit is used for sending the task message to the message queue again when the message queue does not successfully receive the task message.
The above apparatus, optionally, further comprises:
the acquisition unit is used for acquiring the task message from the message queue and verifying the validity of the task message after the transmission service node monitors the task message;
a third determining unit, configured to determine, if the task message passes validity verification, a first storage system and a second storage system corresponding to the transmission task, where the first storage system is a source storage system that stores the target file, and the second storage system is a to-be-transmitted storage system corresponding to the target file;
a judging unit, configured to judge whether the target file exists in the second storage system;
and the writing unit is used for reading the target file from the first storage system and writing the read target file into the second storage system if the target file does not exist in the second storage system.
The above apparatus, optionally, further comprises:
and the ending unit is used for ending the execution process of the transmission task if the target file exists in the second storage system.
The above apparatus, optionally, further comprises: after the task message is sent to a message queue, the task state of the transmission task is updated to be executed;
the update unit includes:
the judging subunit is used for determining the current task state of the transmission task and judging whether the current task state of the transmission task is under execution;
the updating subunit is configured to update the task state of the transmission task to be successful in execution or failed in execution according to the execution result of the transmission task if the current task state of the transmission task is in execution;
and the recording subunit is used for recording the abnormal information if the current task state of the transmission task is not the executing state.
The above apparatus, optionally, further comprises:
and the query unit is used for determining the current task state of the transmission task when receiving the transmission query request of the target file sent by the service system, generating a corresponding query result according to the current task state of the transmission task, and sending the query result to the service system.
A storage medium, the storage medium comprising stored instructions, wherein when the instructions are executed, a device on which the storage medium is located is controlled to execute the file transfer method.
An electronic device comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the file transfer method described above.
Based on the file transmission method provided by the embodiment of the invention, the file transmission method comprises the following steps: when a transmission request of a target file sent by a service system is received, determining transmission information corresponding to the target file; creating a transmission task corresponding to the target file based on the transmission information, and initializing a task state of the transmission task; determining a task message corresponding to the transmission task, and sending the task message to a pre-established message queue, so that a transmission service node which monitors the message queue in real time executes the transmission task after monitoring the task message, determines an execution result of the transmission task after completing an execution process of the transmission task, and sends the execution result of the transmission task to the message queue; and when the message queue receives the execution result of the transmission task, acquiring the execution result of the transmission task from the message queue, updating the task state of the transmission task according to the execution result of the transmission task, and finishing the transmission process of the target file. By applying the method provided by the embodiment of the invention, when the service system needs to transmit the file, only the transmission request needs to be sent, and when the main body executing the method receives the transmission request sent by the service system, the file transmission process corresponding to the transmission request can be completed. The file transmission by the service system is not required, the complex transmission process is not required to be processed, a large amount of time and resources consumed in the operation process of the service system can be avoided for file transmission, and the load of the service system can be effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for transmitting a file according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a file transfer system according to an embodiment of the present invention;
fig. 3 is a flowchart of another method of a file transmission method according to an embodiment of the present invention;
fig. 4 is a signaling flowchart of a file transmission method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a file transfer device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
As can be seen from the background art, in the prior art, file transmission between service systems generally includes file transmission performed by a service system that needs to transmit a file, where the service system needs to process a complex transmission flow, so that the efficiency of the service system in transmitting the file is low, and therefore the service system needs to consume more resources and time to process a task of file transmission, so that the load of the service system is large.
Therefore, the embodiment of the present invention provides a file transmission method, which can call a file transmission service to complete file transmission by sending a transmission request when a service system needs to transmit a file, and does not need the service system to transmit the file by itself, thereby reducing the workload of the service system and effectively reducing the load of the service system.
The embodiment of the invention provides a file transmission method, which can be applied to a file transmission system, and an execution main body of the file transmission method can be a server of the file transmission system. The method flow chart of the method is shown in fig. 1, and comprises the following steps:
s101: when a transmission request of a target file sent by a service system is received, determining transmission information corresponding to the target file;
in the method provided by the embodiment of the invention, a file transmission system can be constructed in advance by adopting a micro-service architecture mode and is used for processing file transmission among all service systems. The micro-service architecture is a method for developing a single application program as a set of small services, each service is cohesive enough, codes are easy to understand, development efficiency is improved, the services can be independently deployed, and each service can be expanded independently without mutual influence.
At least one transmission scheduling service node, a message queue and at least one transmission service node are deployed in the file transmission system. The transmission scheduling service node can communicate with each service system, and each service system can call a pre-configured API interface to call the service and send a transmission request to the transmission scheduling service node. The message queue can adopt a kafka message queue, which is a high-throughput distributed publish-subscribe message system. And the transmission service node is used for butting each storage system corresponding to each service system and executing a transmission task. The transmission service node supports file transmission operations such as reading and writing on various storage systems, and can execute corresponding transmission operations according to the system type related to the transmission task.
In the method provided by the embodiment of the present invention, when a service system needs to transmit a target file stored in a corresponding storage system to another service system (target service system) in an operation process, a transmission request of the target file may be generated, and an interface is called to send the transmission request to a transmission scheduling service node to request that the target file is transmitted to the storage system corresponding to the target service system.
When the transmission scheduling service node receives a transmission request sent by the service system, the transmission request may be analyzed, and transmission information of a target file to be transmitted is obtained from the transmission request, where the transmission information includes a storage location of the target file and a storage system of the target file to be transmitted, for example, a storage location of the target file, and when the storage system is a Hadoop distributed file system HDFS, an absolute path of the target file is included in the transmission information. When the Storage system is a Cloud Object Storage (COS), the transmission information includes a region, a bucket name, a Storage path, and the like corresponding to the target file. The storage location of the target file is also typically the storage system corresponding to the business system that sent the transfer request.
S102: creating a transmission task corresponding to the target file based on the transmission information, and initializing a task state of the transmission task;
in the method provided by the embodiment of the invention, the transmission scheduling service node can create the transmission task according to the transmission information, after the transmission task corresponding to the target file is created, the task information of the transmission task is stored in the database, and the task state of the transmission task is initialized to the state to be executed. Meanwhile, after the transmission task corresponding to the target file is successfully created, response information of successful task creation can be sent to the service system sending the transmission request.
S103: determining a task message corresponding to the transmission task, and sending the task message to a pre-established message queue, so that a transmission service node which monitors the message queue in real time executes the transmission task after monitoring the task message, determines an execution result of the transmission task after completing an execution process of the transmission task, and sends the execution result of the transmission task to the message queue;
in the method provided by the embodiment of the present invention, optionally, for the transmission task stored in the database, the transmission scheduling service node may encapsulate the task information of the transmission task into a corresponding message in real time, obtain a task message corresponding to the transmission task, and send the task message to a pre-established message queue.
Optionally, for the transmission task stored in the database, the transmission scheduling service node may also scan the transmission task in the to-be-executed state in the database at regular time, encapsulate the task information of the transmission task in the to-be-executed state into a corresponding task message, send the corresponding task message to the message queue, and update the task state of the transmission task in the database to the executing state after the task message is sent to the message queue.
In the method provided by the embodiment of the invention, the transmission service node can monitor whether the message queue in the system receives the task message sent by the transmission scheduling service node in real time. After monitoring that a new task message is received in the message queue, the transmission service node can acquire the task message from the message queue and execute a transmission task corresponding to the task message, and according to a source storage system corresponding to a target file in the transmission task and a storage system type corresponding to a target storage system of the target file to be transmitted, the transmission service node can execute corresponding transmission and storage operations and write the target file in the source storage system into the target storage system. After the transmission service node completes the execution process of the transmission task, the execution result of the transmission task can be determined according to the operation result of the transmission storage operation, and the execution result is sent to the message queue.
S104: and when the message queue receives the execution result of the transmission task, acquiring the execution result of the transmission task from the message queue, updating the task state of the transmission task according to the execution result of the transmission task, and finishing the transmission process of the target file.
In the method provided by the embodiment of the present invention, the transmission scheduling serving node may also monitor the message queue in real time, and when it is monitored that the execution result of the transmission task sent by the transmission serving node is received in the message queue, the execution result may be obtained from the message queue, and the task state of the transmission task in the database is updated according to the execution result to represent that the execution of the transmission task fails or is successful, so as to end the transmission process of the target file corresponding to the transmission request. The updated task state is also stored in the database for the service system to inquire the transmission result or actively feed back the transmission result to the service system.
Based on the method provided by the embodiment of the invention, when a service system in a network needs to transmit a file to another service system, a preset interface can be called, a transmission request of a target file is sent to a transmission scheduling service node in a pre-constructed file transmission system, and the file transmission system is triggered to complete the transmission process of the target file. By applying the method provided by the embodiment of the invention, the service system calls the preset interface to send the transmission request, and triggers the file transmission system to finish the file transmission operation. The service system can process other service requests after sending the requests without the self transmission operation of the service system, thereby realizing an asynchronous transmission mode, occupying no processing resource of the service system, reducing the load and workload of the service system and increasing the service processing capacity of the service system. Meanwhile, the development complexity of each service system can be reduced, and the development efficiency of the service systems can be improved.
In addition, in the method provided by the embodiment of the invention, the file transmission system is constructed in a micro-service mode to complete file transmission, and when a new storage system is added in the network, corresponding configuration is carried out in the file transmission system without changing the calling interface of each service system. On the other hand, the message queue can decouple the system, provide data redundancy, facilitate the expansion of the system, have good peak clipping effect, and can avoid the system instability caused by the sudden change of the flow.
To better illustrate the method provided by the embodiment of the present invention, an embodiment of the present invention provides another file transmission method, and fig. 2 shows a schematic structural diagram of a file transmission system provided by the embodiment of the present invention, and based on the method shown in fig. 1, in the method provided by the embodiment of the present invention, a file transmission system adopting a stateless micro service architecture includes a transmission scheduling service node deployed in a cluster, a message queue, and a transmission service node deployed in a cluster.
In the method provided by the embodiment of the present invention, for the clustered deployment of each transmission scheduling service node, each transmission scheduling service node can be configured to each service system in the network, that is, each service system can call a preset interface to send a transmission request to its corresponding transmission scheduling service node when needing to transmit a file, and can also configure each transmission scheduling service node according to a load balancing principle, so that the service system needing to transmit a file sends a transmission request to a currently idle transmission scheduling service node.
For each transmission service node deployed in a cluster, each task message in the message queue is only acquired once in the process of monitoring the message queue by each transmission service node, that is, only one transmission service node executes a certain transmission task, and after the task message in the message queue is acquired, the task message can be identified so as to avoid repeated execution of the task due to re-acquisition. In a specific implementation process, each transmission service node may obtain task messages from a message queue in order, that is, when a new task message is received in the message queue, the task message is obtained by a current node determined by the order, and when a next task message is received, the task message is obtained by a next node. Or when monitoring the task message, the idle transmission service node acquires the task message. It should be noted that the specific manner of allocating the task message in each transmission service node does not affect the method implementation function provided by the embodiment of the present invention.
When the transmission scheduling service node monitors the message queue to obtain the execution result of the transmission task, similarly, the execution result of each transmission task in the message queue can be obtained once, so as to avoid the occurrence of abnormality when the task state of the transmission task in the database is updated. Each transmission scheduling service node can only monitor the execution result of the transmission task processed by the transmission scheduling service node, and update the task state corresponding to the transmission task in the database.
In fig. 2, the transmission service node cluster may interface a plurality of storage systems (file storage), each storage system (file storage) corresponds to each service system in actual application, that is, is used for storing a file of the service system corresponding to the storage system, and each transmission service node may perform read-write operation on each interfaced file storage. Due to the requirements of actual data processing and storage, different types of storage media are usually adopted by each service system, that is, different types of storage systems are corresponded to. Each transmission service node supports read-write flow of various storage media. In practical storage media, three ways of block storage, file storage and object storage are common, and are briefly introduced here.
The block storage is directly hung on the host just like a hard disk, is generally used for storing a direct storage space of the host and database application, provides service to the outside at a physical layer, and belongs to a storage mode of a comparison bottom layer. The file Storage is different from the block Storage at the lower layer, and is raised to an application layer, generally referred to as NAS (Network Attached Storage), which is a set of Network Storage devices, and is accessed through TCP/IP, and the protocol is NFSv3/v 4. The method is generally used for sharing data by a plurality of cloud servers, such as server log centralized management, office file sharing and the like. The object storage has the characteristics of high speed of block storage, sharing of file storage and the like, is intelligent, has a CPU, a memory, a network and a disk, is higher than the characteristics of block storage and file storage, and a cloud service provider generally provides RestAPI for uploading, downloading and reading user files so as to integrate the services.
It should be noted that the system architecture shown in fig. 2 is only an embodiment provided for better illustrating the method of the present invention, and in a specific implementation process, the transmission scheduling service node or the transmission service node may not be deployed in a clustered manner, so that the implementation function of the method provided by the embodiment of the present invention is not affected.
Based on the method provided by the embodiment of the invention, each working node is deployed in a cluster mode, and a stateless micro-service architecture is adopted, so that the capacity can be rapidly expanded when a plurality of tasks are met, and the transmission processing capacity is improved. When the number of tasks is small, partial resources can be released, resource waste is reduced, and the resource utilization rate of the network is improved.
Further, on the basis of the method shown in fig. 1, in the method provided in the embodiment of the present invention, after the task message is sent to the message queue in step S103, when the message queue does not successfully receive the task message, the task message is sent to the message queue again.
In the method provided by the embodiment of the invention, after the transmission scheduling service node sends the task message corresponding to the transmission task to the message queue, the transmission scheduling service node can monitor whether the message queue successfully receives the task message, and if the task message is not successfully pushed to the message queue, namely the message queue does not successfully receive the task message, the transmission scheduling service node can send the task message to the message queue again.
Based on the method provided by the embodiment of the invention, a failure retry mechanism is added in the pushing process of the task message, and when the message transmission fails caused by a network or other uncontrollable factors, the corresponding task message can be sent again, so that the reliability of the system is improved.
To better describe the method provided by the embodiment of the present invention, in combination with the flowchart shown in fig. 3, an embodiment of the present invention provides another file transmission method, on the basis of the method shown in fig. 1, in the method provided by the embodiment of the present invention, as mentioned in step S103, after the transmission service node monitors the task message, the process of executing the transmission task includes:
s201: after the transmission service node monitors the task message, acquiring the task message from the message queue, and verifying the validity of the task message;
in the method provided by the embodiment of the invention, when a transmission service node of a message queue is monitored in real time and the task message is received in the message queue, the transmission service node can judge whether an idle thread exists in a corresponding thread pool at present, if so, the task message is obtained from the message queue, and the validity of the task message is verified. The validity verification is mainly to check whether the parameters of the task message are complete and the format is correct or not through a preset verification mode.
If no idle thread exists in the thread pool corresponding to the transmission service node, the task message can not be acquired temporarily.
S202: if the task message passes validity verification, determining a first storage system and a second storage system corresponding to the transmission task, wherein the first storage system is a source storage system for storing the target file, and the second storage system is a to-be-transmitted storage system corresponding to the target file;
in the method provided by the embodiment of the invention, if each content of the task message meets the validity requirement, one thread can be taken out from the thread pool through the validity verification of the task message, and the transmission task corresponding to the task message is executed. And determining a source storage system of the target file to be transmitted and a storage system to be transmitted corresponding to the target file, namely the storage system to which the target file needs to be transmitted and the storage system corresponding to the target service system, according to the task information of the transmission task.
Optionally, if the task message fails to pass the validity verification, determining verification error information, encapsulating the verification error information into a corresponding verification message, and sending the verification message to a message queue.
S203: judging whether the target file exists in the second storage system or not;
in the method provided by the embodiment of the invention, the transmission service node can inquire whether the storage system stores the target file or not by accessing the second storage system to which the target file is currently transmitted.
S204: and if the target file does not exist in the second storage system, reading the target file from the first storage system, and writing the read target file into the second storage system.
In the method provided by the embodiment of the invention, if the second storage system has no target file after judgment, the first storage system, namely the source storage system of the target file, is accessed, the target file is read from the source storage system, and after the file is read, the second storage system is accessed, the read target file is written into the second storage system, so that the second storage system stores the target file, and the target file is transmitted to the second storage system. And after the write-in operation is executed, generating an execution result corresponding to the transmission task according to the write-in result, and sending the execution result to a message queue for the transmission scheduling service node to obtain.
S205: and if the target file exists in the second storage system, ending the execution process of the transmission task.
In the method provided by the embodiment of the invention, if the second storage system already stores the target file after judgment, the task transmission success message can be directly encapsulated as the execution result of the transmission task, and the execution result is sent to the message queue for the transmission scheduling service node to obtain.
Based on the method provided by the embodiment of the invention, on one hand, the transmission service node can carry out validity verification on the task message in the process of executing the transmission task, and the influence on task execution caused by receiving the abnormal task message is avoided. On the other hand, before the read-write operation is executed, whether the target file is stored in the storage system of the file to be transmitted or not can be judged, the read-write operation is performed after the target file is determined not to be stored in the storage system, when the target file is stored in the storage system, the transmission is regarded as successful, the transmission process is finished, the transmission is not repeated, the system resources can be saved, the task processing efficiency is improved, and meanwhile, the storage abnormity caused by the repeated transmission operation can be avoided.
Further, on the basis of the method shown in fig. 1, in step S103 of the method provided in the embodiment of the present invention, after sending the task message to a message queue, the task state of the transmission task is updated to the executing state.
Thereafter, the process of updating the task state of the transmission task according to the execution result of the transmission task, which is mentioned in step S104, includes:
determining the current task state of the transmission task, and judging whether the current task state of the transmission task is being executed;
in the method provided by the embodiment of the invention, after the transmission scheduling service node sends the task message to the message queue, the task state corresponding to the transmission task in the database can be updated to be the executing state. After the transmission scheduling service node obtains the execution result of the transmission task, the current task state of the transmission task can be obtained from the database, and whether the current task state is the executing state is judged, so as to judge whether the task state of the transmission task is in the normal state.
In the method provided by the embodiment of the present invention, in the normal execution process, the task state of the transmission task should be in the executing state after the task message is sent until the update is performed according to the execution result, so that whether the state is normal is determined by determining whether the task state is in the executing state.
If the current task state of the transmission task is in execution, updating the task state of the transmission task to be successful in execution or failed in execution according to the execution result of the transmission task;
in the method provided by the embodiment of the present invention, if the current task state of the transmission task is the executing state, the task state of the transmission task is updated according to the execution result, if the file writing failure is displayed in the execution result, the task state of the transmission task is updated to the executing failure, and if the file writing success is displayed in the execution result, the task state of the transmission task is updated to the executing success.
And if the current task state of the transmission task is not the executing state, recording abnormal information.
In the method provided by the embodiment of the present invention, if the current task state of the transmission task is not being executed, but is executed successfully, executed unsuccessfully, or in another task state, the abnormal information is recorded.
Based on the method provided by the embodiment of the invention, whether the transmission task is normally executed or not can be judged when the task state of the transmission task is updated, and when the transmission task is abnormally executed repeatedly and the like due to the unstable network or the restart of a certain service in the system and the like, the abnormal condition can be recorded when the task state is updated, thereby being beneficial to the follow-up check.
In order to better describe the method provided by the embodiment of the present invention, the embodiment of the present invention provides another file transmission method, and on the basis of the method shown in fig. 1, the method provided by the embodiment of the present invention further includes:
when receiving a transmission query request of the target file sent by the service system, determining the current task state of the transmission task, generating a corresponding query result according to the current task state of the transmission task, and sending the query result to the service system.
In the method provided by the embodiment of the invention, the transmission scheduling service node is also configured to respond to the transmission result query of the service system. Each service system can call a task query interface provided by the transmission scheduling service node to query the execution result of the target file in a timing or condition triggering mode.
In the method provided by the embodiment of the present invention, in step S102, the transmission scheduling service node may send response information indicating that the task creation is successful to the service system, where the response information includes transmission task information of the transmission task, such as a task ID and a task state.
When the service system receives the transmission task information returned by the transmission scheduling service node, the information can be stored for querying the execution result of the transmission task.
When the business system triggers the query process, a preset task query interface can be called, and a transmission query request of the target file is sent to the transmission scheduling service node based on the transmission task information. The query process of the service system can be triggered by a preset time point in a timed manner, can be triggered by the active query operation of a user of the service system, and can also be triggered by the service process which focuses on the execution result.
When receiving the transmission query request, the transmission scheduling service node may obtain the current task state of the transmission task from the database, generate a corresponding query result, and feed back the query result to the service system. When the task state of the transmission task is in a state to be executed or in execution, an inquiry result indicating that the transmission task is completed and indicating that the transmission is failed or successful may be generated.
Based on the method provided by the embodiment of the invention, the service system can also inquire the transmission result of the target file by calling the preset task inquiry interface, so that the service system can conveniently obtain the transmission result and carry out the next service operation according to the transmission result.
In order to better improve the working process of the file transmission system in the method provided by the embodiment of the present invention, on the basis of the method provided by the above embodiment, fig. 4 shows a signaling flowchart of another file transmission method provided by the embodiment of the present invention.
The file transmission system provided by the embodiment of the invention can be called by each service system with a preset calling interface, and the file transmission system can simultaneously process each transmission request sent by each service system.
The flow chart shown in fig. 4 is only a schematic description, and the service system shown in the figure is characterized in that the service system side actually corresponds to a plurality of service systems, and two transmission requests are sent by different service systems.
In the method provided by the embodiment of the present invention, the storage system corresponding to the service system 1 is the file storage 1, the storage system corresponding to the service system 2 is the file storage 2, and the storage system corresponding to the service system 3 is the file storage 3.
During operation of business system 1, a need arises for file 1 to be transferred to business system 3, while business system 2 arises a need for file 2 to be transferred to business system 1. At this time, the service system 1 sends a transmission request for file 1 interaction to the transmission scheduling service, and the service system 2 sends a transmission request for file 2 interaction to the transmission scheduling service. The transmission scheduling service creates and stores the task corresponding to each transmission request, and then sends the task to the file storage transmission service, so that the file storage transmission service executes corresponding read-write operation and feeds back the result, and the transmission scheduling service updates the result of each task for the query of the service system. The specific processing flow of each task can be referred to the description of the above embodiment, and is not described herein again.
It should be noted that the processing procedures of each transmission request of the file transmission system are independent from each other, and in the processing procedures corresponding to the two tasks shown in fig. 4, there is no specific precedence relationship between signals in different processing procedures.
Corresponding to the file transmission method shown in fig. 1, an embodiment of the present invention further provides a file transmission apparatus, which is used for implementing the method shown in fig. 1 specifically, and a schematic structural diagram of the file transmission apparatus is shown in fig. 5, where the file transmission apparatus includes:
a first determining unit 301, configured to determine, when a transmission request of a target file sent by a service system is received, transmission information corresponding to the target file;
a creating unit 302, configured to create a transmission task corresponding to the target file based on the transmission information, and initialize a task state of the transmission task;
a second determining unit 303, configured to determine a task message corresponding to the transmission task, and send the task message to a pre-established message queue, so that a transmission service node that monitors the message queue in real time executes the transmission task after monitoring the task message, and after completing an execution process of the transmission task, determines an execution result of the transmission task, and sends the execution result of the transmission task to the message queue;
an updating unit 304, configured to, when the message queue receives the execution result of the transmission task, obtain the execution result of the transmission task from the message queue, update the task state of the transmission task according to the execution result of the transmission task, and end the transmission process of the target file.
By applying the device provided by the embodiment of the invention, the service system calls the preset interface to send the transmission request, and triggers the file transmission system to finish the file transmission operation. The service system can process other service requests after sending the requests without the self transmission operation of the service system, thereby realizing an asynchronous transmission mode, occupying no processing resource of the service system, reducing the load and workload of the service system and increasing the service processing capacity of the service system. Meanwhile, the development complexity of each service system can be reduced, and the development efficiency of the service systems can be improved.
Further, on the basis of the above apparatus, the apparatus provided in the embodiment of the present invention further includes:
and the sending unit is used for sending the task message to the message queue again when the message queue does not successfully receive the task message.
Further, on the basis of the above apparatus, the apparatus provided in the embodiment of the present invention further includes:
the acquisition unit is used for acquiring the task message from the message queue and verifying the validity of the task message after the transmission service node monitors the task message;
a third determining unit, configured to determine, if the task message passes validity verification, a first storage system and a second storage system corresponding to the transmission task, where the first storage system is a source storage system that stores the target file, and the second storage system is a to-be-transmitted storage system corresponding to the target file;
a judging unit, configured to judge whether the target file exists in the second storage system;
and the writing unit is used for reading the target file from the first storage system and writing the read target file into the second storage system if the target file does not exist in the second storage system.
Further, on the basis of the above apparatus, the apparatus provided in the embodiment of the present invention further includes:
and the ending unit is used for ending the execution process of the transmission task if the target file exists in the second storage system.
Further, on the basis of the above apparatus, the apparatus provided in the embodiment of the present invention further includes: after the task message is sent to a message queue, the task state of the transmission task is updated to be executed;
the updating unit 304 includes:
the judging subunit is used for determining the current task state of the transmission task and judging whether the current task state of the transmission task is under execution;
the updating subunit is configured to update the task state of the transmission task to be successful in execution or failed in execution according to the execution result of the transmission task if the current task state of the transmission task is in execution;
and the recording subunit is used for recording the abnormal information if the current task state of the transmission task is not the executing state.
Further, on the basis of the above apparatus, the apparatus provided in the embodiment of the present invention further includes:
and the query unit is used for determining the current task state of the transmission task when receiving the transmission query request of the target file sent by the service system, generating a corresponding query result according to the current task state of the transmission task, and sending the query result to the service system.
The embodiment of the invention also provides a storage medium, which comprises stored instructions, wherein when the instructions are executed, the equipment where the storage medium is located is controlled to execute the file transmission method.
An electronic device is provided in an embodiment of the present invention, and the structural diagram of the electronic device is shown in fig. 6, which specifically includes a memory 401 and one or more instructions 402, where the one or more instructions 402 are stored in the memory 401 and configured to be executed by one or more processors 403 to perform the following operations for executing the one or more instructions 402:
when a transmission request of a target file sent by a service system is received, determining transmission information corresponding to the target file;
creating a transmission task corresponding to the target file based on the transmission information, and initializing a task state of the transmission task;
determining a task message corresponding to the transmission task, and sending the task message to a pre-established message queue, so that a transmission service node which monitors the message queue in real time executes the transmission task after monitoring the task message, determines an execution result of the transmission task after completing an execution process of the transmission task, and sends the execution result of the transmission task to the message queue;
and when the message queue receives the execution result of the transmission task, acquiring the execution result of the transmission task from the message queue, updating the task state of the transmission task according to the execution result of the transmission task, and finishing the transmission process of the target file.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for file transfer, comprising:
when a transmission request of a target file sent by a service system is received, determining transmission information corresponding to the target file;
creating a transmission task corresponding to the target file based on the transmission information, and initializing a task state of the transmission task;
determining a task message corresponding to the transmission task, and sending the task message to a pre-established message queue, so that a transmission service node which monitors the message queue in real time executes the transmission task after monitoring the task message, determines an execution result of the transmission task after completing an execution process of the transmission task, and sends the execution result of the transmission task to the message queue;
and when the message queue receives the execution result of the transmission task, acquiring the execution result of the transmission task from the message queue, updating the task state of the transmission task according to the execution result of the transmission task, and finishing the transmission process of the target file.
2. The method of claim 1, wherein after sending the task message to a message queue, further comprising:
and when the task message is not successfully received by the message queue, sending the task message to the message queue again.
3. The method of claim 1, wherein the transmission service node performs the process of transmitting the task after listening to the task message, and wherein the process comprises:
after the transmission service node monitors the task message, acquiring the task message from the message queue, and verifying the validity of the task message;
if the task message passes validity verification, determining a first storage system and a second storage system corresponding to the transmission task, wherein the first storage system is a source storage system for storing the target file, and the second storage system is a to-be-transmitted storage system corresponding to the target file;
judging whether the target file exists in the second storage system or not;
and if the target file does not exist in the second storage system, reading the target file from the first storage system, and writing the read target file into the second storage system.
4. The method of claim 3, further comprising:
and if the target file exists in the second storage system, ending the execution process of the transmission task.
5. The method of claim 1, wherein after sending the task message to a message queue, updating a task status of the transfer task to be executing; the process of updating the task state of the transmission task according to the execution result of the transmission task includes:
determining the current task state of the transmission task, and judging whether the current task state of the transmission task is being executed;
if the current task state of the transmission task is in execution, updating the task state of the transmission task to be successful in execution or failed in execution according to the execution result of the transmission task;
and if the current task state of the transmission task is not the executing state, recording abnormal information.
6. The method of claim 1, further comprising:
when receiving a transmission query request of the target file sent by the service system, determining the current task state of the transmission task, generating a corresponding query result according to the current task state of the transmission task, and sending the query result to the service system.
7. A file transfer apparatus, comprising:
the system comprises a first determining unit, a second determining unit and a third determining unit, wherein the first determining unit is used for determining transmission information corresponding to a target file when a transmission request of the target file sent by a service system is received;
the creating unit is used for creating a transmission task corresponding to the target file based on the transmission information and initializing the task state of the transmission task;
a second determining unit, configured to determine a task message corresponding to the transmission task, and send the task message to a pre-established message queue, so that a transmission service node that monitors the message queue in real time executes the transmission task after monitoring the task message, and after completing an execution process of the transmission task, determines an execution result of the transmission task, and sends the execution result of the transmission task to the message queue;
and the updating unit is used for acquiring the execution result of the transmission task from the message queue when the message queue receives the execution result of the transmission task, updating the task state of the transmission task according to the execution result of the transmission task, and ending the transmission process of the target file.
8. The apparatus of claim 7, further comprising:
and the sending unit is used for sending the task message to the message queue again when the message queue does not successfully receive the task message.
9. A storage medium comprising stored instructions, wherein the instructions, when executed, control a device on which the storage medium is located to perform a file transfer method according to any one of claims 1 to 6.
10. An electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the file transfer method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011570095.2A CN112559461A (en) | 2020-12-26 | 2020-12-26 | File transmission method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011570095.2A CN112559461A (en) | 2020-12-26 | 2020-12-26 | File transmission method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112559461A true CN112559461A (en) | 2021-03-26 |
Family
ID=75033333
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011570095.2A Pending CN112559461A (en) | 2020-12-26 | 2020-12-26 | File transmission method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559461A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113114757A (en) * | 2021-04-09 | 2021-07-13 | 中国工商银行股份有限公司 | File transmission method, device and equipment |
| CN113364852A (en) * | 2021-06-02 | 2021-09-07 | 中国工商银行股份有限公司 | File transmission method and device and server |
| CN116132309A (en) * | 2023-02-08 | 2023-05-16 | 浪潮通信信息系统有限公司 | Batch business processing method and system for network management resources |
| CN116132309B (en) * | 2023-02-08 | 2024-04-26 | 浪潮通信信息系统有限公司 | Batch business processing method and system for network management resources |
-
2020
- 2020-12-26 CN CN202011570095.2A patent/CN112559461A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113114757A (en) * | 2021-04-09 | 2021-07-13 | 中国工商银行股份有限公司 | File transmission method, device and equipment |
| CN113114757B (en) * | 2021-04-09 | 2022-09-30 | 中国工商银行股份有限公司 | File transmission method, device and equipment |
| CN113364852A (en) * | 2021-06-02 | 2021-09-07 | 中国工商银行股份有限公司 | File transmission method and device and server |
| CN113364852B (en) * | 2021-06-02 | 2023-04-07 | 中国工商银行股份有限公司 | File transmission method and device and server |
| CN116132309A (en) * | 2023-02-08 | 2023-05-16 | 浪潮通信信息系统有限公司 | Batch business processing method and system for network management resources |
| CN116132309B (en) * | 2023-02-08 | 2024-04-26 | 浪潮通信信息系统有限公司 | Batch business processing method and system for network management resources |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10341196B2 (en) | Reliably updating a messaging system | |
| CN108449410B (en) | Message management method, system and related device in cloud platform | |
| CN109491776B (en) | Task arranging method and system | |
| US8938510B2 (en) | On-demand mailbox synchronization and migration system | |
| CN107544783B (en) | Data updating method, device and system | |
| CN110365762B (en) | Service processing method, device, equipment and storage medium | |
| CN111858007A (en) | Task scheduling method and device based on message middleware | |
| CN112559461A (en) | File transmission method and device, storage medium and electronic equipment | |
| US11223522B1 (en) | Context-based intelligent re-initiation of microservices | |
| JP7161560B2 (en) | Artificial intelligence development platform management method, device, medium | |
| CN113382077B (en) | Micro-service scheduling method, micro-service scheduling device, computer equipment and storage medium | |
| US10802896B2 (en) | Rest gateway for messaging | |
| US20180150246A1 (en) | Managing of Storage Resources | |
| CN113206877A (en) | Session keeping method and device | |
| CN114942845A (en) | Cross-cluster resource scheduling method and device | |
| CN112783672A (en) | Remote procedure call processing method and system | |
| US20140143592A1 (en) | Client based high availability method for message delivery | |
| WO2020252724A1 (en) | Log processing method and device, and computer-readable storage medium | |
| US20240089352A1 (en) | Udp message distribution method, udp message distribution apparatus, electronic device and computer readable storage medium | |
| CN112306640A (en) | Container dispensing method, apparatus, device and medium therefor | |
| CN111756800A (en) | Method and system for processing burst flow | |
| CN109688011B (en) | Agent selection method and device based on OpenStack | |
| US10348814B1 (en) | Efficient storage reclamation for system components managing storage | |
| US20230146880A1 (en) | Management system and management method | |
| US20030023775A1 (en) | Efficient notification of multiple message completions in message passing multi-node data processing systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |