CN117201483A - File transmission method, device, electronic equipment and storage medium - Google Patents

File transmission method, device, electronic equipment and storage medium Download PDF

Info

Publication number
CN117201483A
CN117201483A CN202311234748.3A CN202311234748A CN117201483A CN 117201483 A CN117201483 A CN 117201483A CN 202311234748 A CN202311234748 A CN 202311234748A CN 117201483 A CN117201483 A CN 117201483A
Authority
CN
China
Prior art keywords
task
node
file distribution
target
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311234748.3A
Other languages
Chinese (zh)
Inventor
何雪源
陈锐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202311234748.3A priority Critical patent/CN117201483A/en
Publication of CN117201483A publication Critical patent/CN117201483A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The disclosure provides a file transmission method, a file transmission device, electronic equipment and a storage medium, relates to the technical field of computers, and particularly relates to the fields of cloud computing, big data, information flow and the like. The specific implementation scheme is as follows: in response to determining that a target node of the cluster has obtained a file to be transmitted, obtaining information of a target file distribution task configured for the target node, wherein the cluster further comprises a downstream node of the target node, and the information of the target file distribution task comprises a downstream node identifier of the downstream node; and in response to receiving a first execution instruction for executing the target file distribution task, sending the file to be transmitted to the downstream node according to the downstream node identification.

Description

File transmission method, device, electronic equipment and storage medium
Technical Field
The disclosure relates to the technical field of computers, in particular to the fields of cloud computing, big data, information flow and the like, and specifically relates to a file transmission method, a file transmission device, electronic equipment and a storage medium.
Background
In cloud environments, many businesses can operate and manage very large clusters, such as the Hadoop (a distributed system infrastructure) large data clusters of thousands of nodes that are common. In such environments, there is often a need to download a file (deployment package, configuration file, or data) from one machine or external environment to multiple machines within a cluster.
Disclosure of Invention
The disclosure provides a file transmission method, a file transmission device, electronic equipment and a storage medium.
According to an aspect of the present disclosure, there is provided a file transfer method, including: in response to determining that a target node of the cluster has obtained a file to be transmitted, obtaining information of a target file distribution task configured for the target node, wherein the cluster further comprises a downstream node of the target node, and the information of the target file distribution task comprises a downstream node identifier of the downstream node; and in response to receiving a first execution instruction for executing the target file distribution task, sending the file to be transmitted to the downstream node according to the downstream node identification.
According to another aspect of the present disclosure, there is provided a file transfer apparatus including: the task information acquisition module is used for acquiring information of a target file distribution task configured for the target node in response to determining that the target node of the cluster has acquired a file to be transmitted, wherein the cluster also comprises a downstream node of the target node, and the information of the target file distribution task comprises a downstream node identifier of the downstream node; and the file sending module is used for responding to the received first execution instruction for executing the target file distribution task and sending the file to be transmitted to the downstream node according to the downstream node identification.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the file transfer method of the present disclosure.
According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the file transfer method of the present disclosure.
According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, which when executed by a processor, implements the file transfer method of the present disclosure.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 schematically illustrates an exemplary system architecture to which file transfer methods and apparatus may be applied, according to embodiments of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a file transfer method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates an exemplary block diagram of nodes in a cluster according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a schematic diagram of processing logic of a file distribution task according to an embodiment of the present disclosure;
FIG. 5 schematically illustrates an overall flow chart of determining a task execution state of a current task according to an embodiment of the present disclosure;
FIG. 6 schematically illustrates a task state transition diagram for a single file distribution task or a task in a group of file distribution tasks according to an embodiment of the present disclosure;
FIG. 7 schematically illustrates a block diagram of a file transfer device according to an embodiment of the present disclosure; and
FIG. 8 illustrates a schematic block diagram of an example electronic device that may be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing, applying and the like of the personal information of the user all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public order harmony is not violated.
In the technical scheme of the disclosure, the authorization or consent of the user is obtained before the personal information of the user is obtained or acquired.
When a file needs to be downloaded from a machine or an external environment to a plurality of machines in a cluster, the file is generally implemented by separately building a separate source server or service cluster. Specifically, one or more nodes are selected as sources within the cluster, multiple machines that need to download files concurrently download files from these sources, and a fixed central node is needed to monitor the download status of each node.
The inventor finds that in the process of implementing the disclosed concept, the method needs to independently maintain an independent source server, so that the cost is high, and the bandwidth of the source server easily becomes a downloading bottleneck, so that the efficiency of the whole downloading task is low. In addition, a fixed central node is required to be maintained, and various operations in the file transmission process are completed by the central node, so that the central node is easy to become a stability bottleneck of the scheme.
Fig. 1 schematically illustrates an exemplary system architecture to which file transfer methods and apparatuses may be applied according to embodiments of the present disclosure.
It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the file transmission method and apparatus may be applied may include a terminal device, but the terminal device may implement the file transmission method and apparatus provided by the embodiments of the present disclosure without interaction with a server.
As shown in fig. 1, a system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and/or wireless communication links, and the like.
The user may interact with the server 105 via the network 104 using the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages etc. Various communication client applications, such as a knowledge reading class application, a web browser application, a search class application, an instant messaging tool, a mailbox client and/or social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103.
The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (merely an example) providing support for content browsed by the user with the first terminal apparatus 101, the second terminal apparatus 102, the third terminal apparatus 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.
It should be noted that, the file transfer method provided by the embodiments of the present disclosure may be generally performed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the file transfer apparatus provided in the embodiment of the present disclosure may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103.
Alternatively, the file transfer method provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the file transfer device provided in the embodiments of the present disclosure may be generally disposed in the server 105. The file transfer method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105. Accordingly, the file transmitting apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105.
For example, when the user reads the electronic book online, the first terminal device 101, the second terminal device 102 and the third terminal device 103 may acquire target content in the electronic book pointed by the line of sight of the user, and then send the acquired target content to the server 105, and the server 105 analyzes the target content to determine feature information of the target content; predicting the content of interest of the user according to the characteristic information of the target content; and extract content of interest to the user. Or by a server or server cluster capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and/or the server 105, and finally enabling to extract the content of interest to the user.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically illustrates a flow chart of a file transfer method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 to S220.
In operation S210, in response to determining that the target node of the cluster has obtained the file to be transmitted, obtaining information of a target file distribution task configured for the target node, the cluster further including a downstream node of the target node, the information of the target file distribution task including a downstream node identification of the downstream node.
In operation S220, in response to receiving the first execution instruction for executing the target file distribution task, the file to be transmitted is sent to the downstream node according to the downstream node identification.
According to embodiments of the present disclosure, file transfer may characterize the process of downloading a transfer file from one node and sending it to another node. A cluster may include a plurality of nodes. In case any one node in the cluster receives a file to be transmitted, the node may be determined as a target node. The target file distribution task may characterize a task of sending a file to be transmitted from the target node to a node downstream of the target node.
According to an embodiment of the present disclosure, in a case where a file to be transmitted has not been sent to any one node in a cluster, a batch of initial seed nodes may be first selected within the cluster. And then, downloading the file to be transmitted in parallel from a file source of the file to be transmitted, and sending the file to the initial seed node. The initial seed node that received the file to be transmitted may be determined as the target node.
According to an embodiment of the present disclosure, in a case where there is a portion of target nodes in the cluster that have obtained the file to be transferred, each target node may convert to an identity of the seed node to continue to provide for downloading the file to be transferred to a downstream node of the target node. The downstream node that receives the file to be transmitted may continue to be determined as the target node. In this case, the target file distribution task may be executed from any target node, and the point-to-point file transmission is performed in the cluster until all nodes in the cluster obtain the file to be transmitted, and then the task is finished.
According to the embodiment of the disclosure, the distributed decentralized point-to-point file transmission method is realized, and because point-to-point transmission is implemented in the cluster, bandwidth resources in the cluster can be utilized to the maximum, downloading of one file on a plurality of machines is realized, and the file downloading task can be completed quickly.
The method shown in fig. 2 is further described below in connection with the specific examples.
According to an embodiment of the present disclosure, before performing the above operation S210, it is necessary to first determine a target file distribution task of a target node configuration. The method may include: and responding to the received file transmission instruction representing the file to be transmitted to the cluster, and generating a target file distribution task according to the file transmission information represented by the file transmission instruction. The information of the target file distribution task is sent to a second external storage unit connected with each node of the cluster, so that the information of the target file distribution task is sent to the node of the cluster through the second external storage unit.
According to embodiments of the present disclosure, machine nodes in a cluster are named as peers (nodes), machines and machines can be Peer-to-Peer with each other, and there is no centralized node. JobScheduler, task executor and other components can be normally deployed on each Peer, jobScheduler and task executor are designed to be stateless components, and JobScheduler, task executor and other components of different Peers cannot be perceived mutually. The second external memory unit may be used to achieve information synchronization between peers that are not perceivable to each other.
Fig. 3 schematically illustrates an exemplary block diagram of nodes in a cluster according to an embodiment of the disclosure.
As shown in fig. 3, the cluster 300 includes a first Peer 310, a second Peer 320, a third Peer 330, and a fourth Peer 340.JobScheduler 301 may characterize JobScheduler deployed for any one Peer (not shown in the specific structure of each Peer). Corresponding to the first Peer 310, the second Peer 320, the third Peer 330 and the fourth Peer 340, a first TaskExecutor 311, a second TaskExecutor 321, a third TaskExecutor 331 and a fourth TaskExecutor 341 are respectively deployed. The external storage 303 may characterize the second external storage unit described above.
For example, jobScheduler 301 may characterize JobScheduler in first Peer 310. In the case where none of the peers in the cluster 300 obtain a file to be transmitted, and the JobScheduler 301 receives a file transmission instruction that characterizes a file transmission job 302 that distributes the file to be transmitted to the cluster, the JobSchedule301 may, in response to receiving the file transmission instruction, disassemble the file transmission job 302 into a plurality of file distribution tasks, such as a plurality of file distribution tasks 304 in fig. 3, according to file transmission information, and may synchronize the plurality of file distribution tasks 304 to the peers, such as the first Peer 310, the second Peer 320, the third Peer 330, the fourth Peer 340, and the like, through the external storage 303.
According to the embodiment of the disclosure, the generated target file distribution tasks may be multiple, and information of the multiple target file distribution tasks may be all sent to each node of the cluster. For example, the first TaskExecutor 311, the second TaskExecutor 321, the third TaskExecutor 331, and the fourth TaskExecutor 341 may all obtain the above-mentioned multiple file distribution tasks.
According to the embodiments of the present disclosure, the TaskExecutor may perform a corresponding file distribution task in response to receiving an execution instruction for performing the file distribution task. For example, in response to receiving an execution instruction for executing a file distribution task in the second Peer 320, the file distribution task in the second Peer 320 is executed using the second TaskExecutor 321.
Through the embodiment of the disclosure, by configuring the second external storage unit, each node in the cluster can be independent, the decentralization of the cluster can be realized, and flexible lateral expansion can be realized.
According to an embodiment of the present disclosure, a cluster may include a first node and a second node. The file transfer information may include a first node identification of the first node and a second node identification of the second node. The generating the target file distribution task according to the file transmission information characterized by the file transmission instruction may include: and generating a target file distribution task which takes the first node as a source node and the second node as a destination node according to the first node identifier and the second node identifier.
According to an embodiment of the present disclosure, in a case where there are a plurality of nodes in a cluster, one node may be randomly determined as a first node, and another node may be randomly determined as a second node. In the process of generating the target file distribution task, which node of the first node and the second node is the source node and which node is the destination node may be determined at random, and the method is not limited herein.
For example, referring to fig. 3, a first node identifier peer_id of the first Peer 310 may be included in the file transfer information 1 Second node identification Peer_id of second Peer 320 2 Third node identification Peer_id of third Peer 330 3 Fourth node identifier Peer_id of fourth Peer 340 4 . Can be in Peer_id 1 As source node, with Peer_id 3 For the destination node, a target file distribution task is generated. Can be in Peer_id 2 As source node, with Peer_id 4 For the destination node, another target file distribution task is generated.
It should be noted that, in the case of determining the target file distribution task based on the above manner, since the specific storage address where the file to be transmitted is sent to each node is not defined, the pre-storage address may be predefined in each node, so that the file to be transmitted is stored to the pre-storage address when the node obtains the file to be transmitted.
According to embodiments of the present disclosure, the file transfer information may further include a target storage address of the file to be transferred. The generating the target file distribution task according to the file transmission information characterized by the file transmission instruction may further include: and generating a target file distribution task which takes the first node as a source node, the second node as a target node and the target storage address in the second node as a file storage address of the file to be transmitted according to the first node identification, the second node identification and the target storage address.
According to the embodiment of the disclosure, since the obtained target file distribution task includes the target storage address of the file to be transmitted, the file to be transmitted can be directly stored to the target storage address of the destination node in the process of transmitting the file to be transmitted from the source node to the destination node.
According to an embodiment of the present disclosure, the foregoing node that sends the target file distribution task to the cluster may include: and according to the source node identification of the source node of the target file distribution task, the target file distribution task is sent to the node corresponding to the source node identification.
According to the embodiment of the disclosure, after determining a plurality of target file distribution tasks, tasks transmitted to other nodes with a current node as a source may be filtered out from the plurality of target file distribution tasks according to source node identifications of the target file distribution tasks. The class of tasks is then sent to the current node.
For example, referring to FIG. 3 and the foregoing embodiments, for the use of Peer_id l As source node, with Peer_id 3 The target file distribution task for the destination node may be sent only to the first Peer 310. For the Peer_id 2 As source node, with Peer_id 4 The target file distribution task for the destination node may be sent only to the second Peer 320.
Through the embodiment of the invention, the data volume of the nodes in the cluster can be reduced, and the load of the nodes is lightened.
According to the embodiment of the present disclosure, the JobScheduler, as a distributing part of the file transfer job, needs to perceive the completion of the file transfer job according to the status of the file distribution job. Thus, the TaskExecutor can synchronize the state of the file distribution tasks to the JobScheduler through external storage at regular time.
Based on this, the above file transfer method may further include: and determining target task execution state information of the target file distribution task in response to determining that the current time reaches the predefined time. The target task execution state information is sent to a first external storage unit connected with each node of the cluster, so that the target task execution state information is sent to each node of the cluster through the first external storage unit.
According to an embodiment of the present disclosure, whether the current time reaches the predefined time may be determined by performing a timing operation in advance. The first external memory unit may have the same or similar technical features as the second external memory unit, and will not be described herein.
According to embodiments of the present disclosure, the target task execution state information may include any one of the following: init, waiting, running, fail, done. Init may represent a task initialization that has not been claimed by TaskExecutor. waiting may indicate that the TaskExecutor has claimed the task but is waiting for the upstream task to execute. Running may indicate that TaskExecutor is performing the task, transmitting the file to other nodes. Done may indicate that the file transfer is complete and that the task was performed successfully. Fail may indicate a file transfer failure, a task execution failure, but because of the retry mechanism, the task does not end. Faildone may indicate that the task failed and reached a maximum number of retries, and that the task ended with failed state execution.
It should be noted that the file distribution task may have a fixed upstream task. If and only if the upstream task is successfully executed and the file is successfully downloaded to the node, the node can execute the file distribution task taking the node as a source and continuously distribute the file to other nodes.
Through the embodiment of the invention, the target task execution state information in each node can be updated at regular time, and the cluster can operate stably. In addition, all state synchronization is realized through external storage, so that flexible transverse expansibility can be realized.
According to embodiments of the present disclosure, a TaskExecutor may include two phases when performing one file distribution task. The first phase is to synchronize the upstream task state. The task executor needs to wait until the execution of the upstream task of the current file distribution task is completed, and the task executor can enter a second stage to execute the current task.
Fig. 4 schematically illustrates a schematic diagram of processing logic of a file distribution task according to an embodiment of the present disclosure.
As shown in fig. 4, for example, a first file distribution task 401 may characterize a task of an upstream node of a second file distribution task 402 and a third file distribution task 403. The second file distribution task 402 may characterize the tasks of upstream nodes of the fourth file distribution task 404 and the fifth file distribution task 405. The third file distribution task 403 may characterize the tasks of the upstream nodes of the sixth file distribution task 406 and the seventh file distribution task 407. For these file distribution tasks, only after the task execution of each upstream node is successful, the file distribution task of the child node under each upstream node may be executed.
According to embodiments of the present disclosure, the cluster may also include an upstream node of the target node. The determining the target task execution state information of the target file distribution task may include: and acquiring the execution state information of the upstream task of the upstream file distribution task configured for the upstream node. In response to determining that the upstream task execution state information indicates that the non-execution is complete, the upstream task execution state information is read again after a preset time interval. In response to determining that the upstream task execution state information indicates successful execution, the target task execution state information is updated to being executed.
According to embodiments of the present disclosure, init, waiting, running, fail may each indicate that completion is not performed. done may indicate that execution was successful.
For example, referring to fig. 4, in the case where it is determined that the state of the first file distribution task 401 is any one of init, waiting, running, fail, at least one of TaskExecutor for controlling the second file distribution task 402 and TaskExecutor for controlling the third file distribution task 403 may go to check the state of the first file distribution task 401 again after a time of a preset time interval by sleep until the execution of the first file distribution task 401 ends.
According to embodiments of the present disclosure, done, faildone may each indicate the end of execution.
For example, referring to fig. 4, in the case where the state of the first file distribution task 401 is done, both the second file distribution task 402 and the third file distribution task 403 may enter a running state, both the state of the second file distribution task 402 and the state of the third file distribution task 403 may be updated to running, and file transfer may be started, and the second file distribution task 402 and the third file distribution task 403 may be executed.
According to an embodiment of the present disclosure, the determining the target task execution state information of the target file distribution task may further include: in response to determining that the upstream task execution state information indicates an execution failure, the target task execution state information is updated to the execution failure.
For example, referring to fig. 4, in the case where the state of the second file distribution task 402 is faildone, which means that the second file distribution task 402 fails to execute, the fourth file distribution task 404 and the fifth file distribution task 405 located downstream of the second file distribution task 402 cannot continue to execute either, and the state of the fourth file distribution task 404 and the state of the fifth file distribution task 405 may be updated to faildone. In this case, the states of the second file distribution task 402, the fourth file distribution task 404, and the fifth file distribution task 405 are all ended in execution in a failure state.
According to an embodiment of the present disclosure, the determining the target task execution state information of the target file distribution task may further include: and in response to determining that the retry number of the target file distribution task is equal to the preset number of times, updating the target task execution state information to be the execution failure.
For example, referring to fig. 4, in the case where the third file distribution task 403 fails to be executed and the number of retries has reached the preset number of times, the status of the third file distribution task 403 may be updated to faildone.
According to an embodiment of the present disclosure, the number of retries described above may be determined by: and generating a second execution instruction for re-executing the target file distribution task in response to the fact that the target file distribution task is determined to fail to execute and the fact that the retry number is smaller than the preset number, and adding 1 to the retry number, wherein the initial value of the retry number is 0.
According to an embodiment of the present disclosure, each document distribution task may have a number of retries corresponding thereto. In the case where each file distribution task is started, the value of the number of retries corresponding to the file distribution task may be initialized to 0.
For example, referring to fig. 4, in the case where the state of the third file distribution task 403 is changed to running, the value of the number of retries corresponding to the third file distribution task 403 may be initialized to 0. In the case where the third file distribution task 403 fails to execute, the value of the number of retries corresponding to the third file distribution task 403 may be added by 1, and a second execution instruction may be generated, and the third file distribution task 403 may be re-executed.
According to an embodiment of the present disclosure, the number of retries may also be determined by: and determining the single execution duration of the target file distribution task according to the starting time and the current time of the target file distribution task. And generating a third execution instruction for re-executing the target file distribution task in response to the fact that the single execution duration reaches the preset duration and the fact that the retry number is smaller than the preset number is determined, and adding 1 to the retry number.
According to embodiments of the present disclosure, a task may cause execution to timeout on a node for some anomaly reason. For this case, the TaskExecutor of the node may write the start time of the task into the task execution state when converting the task from waiting to running. When the task execution time exceeds the set timeout threshold parameter, the TaskExecutor may restart execution of the task and increase the number of retries by 1.
Through the disclosed embodiment, the overtime retry mechanism is provided, so that the problem of overtime execution of the file distribution task can be effectively solved, and the problem that the file transmission work cannot be completed for a long time due to overtime of the task is relieved.
FIG. 5 schematically illustrates an overall flow chart for determining the task execution status of a current task in accordance with an embodiment of the present disclosure.
As shown in fig. 5, the method includes operations S501 to S511.
In operation S501, an upstream task upstreamTask is acquired.
In operation S502, it is determined whether or not upstream task (state of upstream task) = = = ' init ' |waiting ' |running ' |fail ' is true? If yes, executing operation S503; if not, operation S504 is performed.
In operation S503, sleep is performed for a period of time.
In operation S504, it is determined that upstreamtask.status= = 'faildone' is true? If yes, executing operation S505; if not, operation S506 is performed.
In operation S505, the current task state is updated to 'faildone'.
In operation S506, it is determined whether upstreamtask.status= = 'done' holds? If yes, executing operation S507; if not, operation S505 is performed.
In operation S507, file transfer is performed.
In operation S508, it is determined that the execution is successful? If yes, operation S509 is performed; if not, operation S510 is performed.
In operation S509, the current task state is updated to 'done'.
In operation S510, it is determined whether the maximum number of retries is reached? If yes, executing operation S505; if not, operation S511 is performed.
In operation S511, the current task state is updated to 'fail'.
By the embodiment of the disclosure, orderly and correct execution of each file distribution task can be realized, and higher execution efficiency and better file transmission effect are realized.
According to the embodiment of the disclosure, in an actual scenario, there may be a case where multiple file transfers are running at the same time, and in this case, there may be a case where multiple tasks write the same file path at the same time. Corresponding to this situation, the task mutex problem in this scenario needs to be considered.
According to embodiments of the present disclosure, the target file distribution task may include a plurality of file distribution tasks generated at a plurality of times. The determining the target task execution state information of the target file distribution task may include: and distributing the file distribution tasks with the same task execution path to the same file distribution task group according to the task execution path information of each of the plurality of file distribution tasks. And determining the task execution state of the file distribution tasks in the same file distribution task group according to preset rules defined for the same file distribution task group, wherein the preset rules are defined according to a mutual exclusion mechanism.
According to the embodiment of the disclosure, the task execution path may be determined according to a source node, a destination node and a target storage path in the information of the file distribution task. When the source node, the destination node, and the target storage path in the information of the plurality of file distribution tasks are the same, it may be determined that the task execution paths of the plurality of file distribution tasks are the same, and the plurality of file distribution tasks may be allocated to the same file distribution task group.
According to an embodiment of the present disclosure, the preset rules determined based on the mutual exclusion mechanism may include any one of the following: at most, only one running state task, namely rule one, can exist in the same group. If a running state task exists in a group, the running state task generated before the generation time of the running state task in the group is updated to be in an ignored state, which means that execution is skipped, and is abbreviated as a rule two. If the upstream task state of a task is ignored, the true state of the task should be the final execution state of the running task within the same group that triggered the task to change to ignored, abbreviated as rule three.
It should be noted that the foregoing preset rules are merely exemplary embodiments, but are not limited thereto, and may include other preset rules that can be determined based on a mutual exclusion mechanism in the art, which is not limited herein.
According to the embodiment of the disclosure, under the condition that a plurality of file distribution tasks executed on the same Peer write the same target path, the plurality of file distribution tasks are distributed to the same file distribution task group, a preset rule is defined for the same file distribution task group based on a mutual exclusion mechanism, and in the process of determining the task execution state of the file distribution tasks in the same file distribution task group based on the preset rule, the task executor can determine the task execution state of each file distribution task in the same file distribution task group through a mutual exclusion mechanism, so that the accuracy of a final execution result can be ensured.
It should be noted that, the above manner of determining the status of the file distribution task is only used to determine the status of the file distribution task in the group. The status of any single file distribution task outside the file distribution task group may be determined based on the foregoing manner of determining the task execution status of the single file distribution task, which is not described herein.
Through the above embodiment of the present disclosure, the correctness of the final execution result may be ensured through a mutual exclusion mechanism.
According to an embodiment of the present disclosure, the preset rule may include, corresponding to the rule one described above: task execution status information of at most one file distribution task in the same file distribution task group indicates that execution is being performed.
With the above-described embodiments of the present disclosure, when a plurality of file distribution tasks executed on the same Peer are writing the same target path, based on the rule, the problem of simultaneous writing can be avoided.
According to an embodiment of the present disclosure, the same file distribution task group may include a first file distribution task generated at a first time and a second file distribution task generated at a second time. Corresponding to the second rule, determining the task execution state of the file distribution task in the same file distribution task group according to the preset rule defined for the same file distribution task group may include: in response to determining that the first task execution state information of the first file distribution task indicates that execution is occurring and that the first time is later than the second time, the second task execution state information of the second file distribution task is updated to an ignore state that characterizes skipped execution.
According to the embodiment of the present disclosure, in the case where the states of the first file distribution task and the second file distribution task are waiting, the state of the first file distribution task that is newly generated (and at the first time) may be updated to run first according to the first time and the second time. The state of the second file distribution task may then be updated to ignored.
By the embodiment of the disclosure, repeated execution can be avoided, and file transmission errors can be effectively reduced. In addition, in the case of the optical fiber,
according to an embodiment of the present disclosure, corresponding to the third rule, determining, according to the preset rule defined for the same file distribution task group, the task execution state of the file distribution task in the same file distribution task group may further include: in response to determining that the first task execution state information indicates successful execution, the second task execution state information is updated to successful execution. In response to determining that the first task execution state information indicates an execution failure, the second task execution state information is updated to the execution failure.
Fig. 6 schematically illustrates a task state transition diagram of a single file distribution task or a task in a file distribution task group according to an embodiment of the present disclosure.
As shown in fig. 6, init may characterize the initial state after each file distribution task is generated. Based on the task obtaining operation, the task may be obtained by the task executor, and at this time, the state of the file distribution task may be updated to waiting.
In the case where a file distribution task group is created with a file distribution task in waiting state and the task executor judges that there is an updated task in the group, the state of the file distribution task in waiting state may be updated to ignored.
Based on the operations performed by the TaskExecutor schedule, a file distribution task may be started, at which time the state of the file distribution task in the waiting state may be updated to running.
In the case where the file distribution task in the running state is successfully executed, the state of the file distribution task in the running state may be updated to done.
In the case where the file distribution task in the running state fails to execute or times out, the state of the file distribution task in the running state may be updated to fail.
In the case where the TaskExecutor determines that the number of retries of the failed-state file distribution task has run out, the state of the failed-state file distribution task may be updated to failed.
In the case where the task executor determines that the retry number of the file distribution task in the fail state is not used up, the state of the file distribution task in the fail state may be updated to waiting.
In the case where the task executor judges that there is an updated task within the group, the state of the file distribution task in the fail state may be updated to ignored.
Through the embodiment of the disclosure, by introducing the concepts of the new file distribution task state ignored and the file distribution task group and combining the processing of the preset rule, the problem of mutual exclusion of the file distribution tasks can be solved, the introduction of the ignored mechanism can effectively improve the file distribution performance of a hot spot path, particularly the scene when the node is down and restored, and the file distribution performance under the scene can be effectively improved.
According to an embodiment of the present disclosure, the file transmission method of the present disclosure may be applied to a business operation and maintenance large-scale cluster scenario, including batch installation of software on a cluster, update of a service, distribution configuration, and the like. The execution efficiency under such a scene can be effectively improved, and the cluster resources are utilized to the maximum extent to reduce the execution time.
Fig. 7 schematically illustrates a block diagram of a file transfer device according to an embodiment of the present disclosure.
As shown in fig. 7, the file transmission apparatus 700 includes a task information acquisition module 710 and a file transmission module 720.
The task information obtaining module 710 is configured to obtain, in response to determining that the target node of the cluster has obtained the file to be transmitted, information of a target file distribution task configured for the target node, where the cluster further includes a downstream node of the target node, and the information of the target file distribution task includes a downstream node identifier of the downstream node.
The file sending module 720 is configured to send, in response to receiving a first execution instruction for executing the target file distribution task, a file to be transmitted to a downstream node according to the downstream node identifier.
According to an embodiment of the present disclosure, the file transmission device further includes a status determination module and a status information transmission module.
And the state determining module is used for determining target task execution state information of the target file distribution task in response to determining that the current moment reaches the predefined moment.
The state information sending module is used for sending the target task execution state information to a first external storage unit connected with each node of the cluster, so that the target task execution state information is sent to each node of the cluster through the first external storage unit.
According to an embodiment of the present disclosure, the target file distribution task includes a plurality of file distribution tasks generated at a plurality of times. The state determination module includes a task group sub-module and a state determination sub-module.
The task group distribution sub-module is used for distributing the file distribution tasks with the same task execution path to the same file distribution task group according to the task execution path information of each of the plurality of file distribution tasks.
The state determining sub-module is used for determining the task execution state of the file distribution tasks in the same file distribution task group according to preset rules defined for the same file distribution task group, wherein the preset rules are defined according to a mutual exclusion mechanism.
According to an embodiment of the present disclosure, the preset rule includes: task execution status information of at most one file distribution task in the same file distribution task group indicates that execution is being performed.
According to an embodiment of the present disclosure, the same file distribution task group includes a first file distribution task generated at a first time and a second file distribution task generated at a second time. The state determination submodule includes a first state update unit.
And a first state updating unit configured to update second task execution state information of the second file distribution task to an ignore state characterizing skip execution in response to determining that the first task execution state information of the first file distribution task indicates that the first file distribution task is being executed and that the first time is later than the second time.
According to an embodiment of the present disclosure, the state determination sub-module further comprises a second state updating unit and a third state updating unit.
A second state updating unit configured to update the second task execution state information to execution success in response to determining that the first task execution state information indicates execution success; and
and a third state updating unit configured to update the second task execution state information to the execution failure in response to determining that the first task execution state information indicates the execution failure.
According to an embodiment of the present disclosure, the cluster further comprises an upstream node of the target node. The state determining module comprises an upstream state obtaining sub-module, an upstream state reading sub-module and a first state updating sub-module.
And the upstream state acquisition sub-module is used for acquiring the upstream task execution state information of the upstream file distribution task configured for the upstream node.
And the upstream state reading sub-module is used for reading the upstream task execution state information again after a preset time interval in response to determining that the upstream task execution state information indicates that the execution is not completed.
And a first status updating sub-module for updating the target task execution status information to be executing in response to determining that the upstream task execution status information indicates successful execution.
According to an embodiment of the present disclosure, the state determination module further comprises a second state update sub-module.
And a second status updating sub-module for updating the target task execution status information to an execution failure in response to determining that the upstream task execution status information indicates the execution failure.
According to an embodiment of the present disclosure, the state determination module includes a third state update sub-module.
And the third state updating sub-module is used for updating the execution state information of the target task into the execution failure in response to determining that the retry number of the target file distribution task is equal to the preset number.
According to an embodiment of the present disclosure, the file transfer apparatus further includes a first retry sub-module.
The first retry sub-module is used for responding to the fact that the execution of the target file distribution task is failed, the fact that the retry number is smaller than the preset number is determined, a second execution instruction for re-executing the target file distribution task is generated, the retry number is added with 1, and the initial value of the retry number is 0.
According to an embodiment of the present disclosure, the file transfer device further includes a task duration determination sub-module and a second retry sub-module.
The task duration determining submodule is used for determining the single execution duration of the target file distribution task according to the starting time and the current time of the target file distribution task.
And the second retry sub-module is used for generating a third execution instruction for re-executing the target file distribution task and adding 1 to the retry number in response to the fact that the single execution duration reaches the preset duration and the retry number is smaller than the preset number.
According to an embodiment of the disclosure, the file transmission device further includes a task generation module and a task information transmission module.
And the task generating module is used for responding to the received file transmission instruction representing the file to be transmitted to the cluster, and generating a target file distribution task according to the file transmission information represented by the file transmission instruction.
And the task information sending module is used for sending the information of the target file distribution task to a second external storage unit connected with each node of the cluster so as to send the information of the target file distribution task to the node of the cluster through the second external storage unit.
According to an embodiment of the present disclosure, a cluster includes a first node and a second node, and file transfer information includes a first node identification of the first node and a second node identification of the second node. The task generation module includes a first task generation sub-module.
And the first task generation sub-module is used for generating a target file distribution task which takes the first node as a source node and the second node as a destination node according to the first node identifier and the second node identifier.
According to an embodiment of the present disclosure, the file transfer information further includes a target storage address of the file to be transferred. The task generation module further includes a second task generation sub-module.
And the second task generating sub-module is used for generating a target file distribution task which takes the first node as a source node, takes the second node as a target node and takes the target storage address in the second node as a file storage address of a file to be transmitted according to the first node identification, the second node identification and the target storage address.
According to an embodiment of the present disclosure, the task information transmission module includes a task information transmission sub-module.
And the task information sending sub-module is used for sending the target file distribution task to the node corresponding to the source node identifier according to the source node identifier of the source node of the target file distribution task.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the file transfer method of the present disclosure.
According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a file transfer method of the present disclosure.
According to an embodiment of the present disclosure, a computer program product includes a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the file transfer method of the present disclosure when executed by a processor.
Fig. 8 illustrates a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 8, the apparatus 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.
Various components in device 800 are connected to an input/output (I/O) interface 805, including: an input unit 806 such as a keyboard, mouse, etc.; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, or the like. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 801 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the respective methods and processes described above, such as a file transfer method. For example, in some embodiments, the file transfer method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 800 via ROM 802 and/or communication unit 809. When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the file transfer method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the file transfer method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims (33)

1. A file transfer method comprising:
in response to determining that a target node of a cluster has obtained a file to be transmitted, obtaining information of a target file distribution task configured for the target node, wherein the cluster further comprises a downstream node of the target node, and the information of the target file distribution task comprises a downstream node identifier of the downstream node; and
and in response to receiving a first execution instruction for executing the target file distribution task, sending the file to be transmitted to the downstream node according to the downstream node identification.
2. The method of claim 1, further comprising:
determining target task execution state information of the target file distribution task in response to determining that the current time reaches a predefined time; and
and sending the target task execution state information to a first external storage unit connected with each node of the cluster, so as to send the target task execution state information to each node of the cluster through the first external storage unit.
3. The method of claim 2, wherein the target file distribution task comprises a plurality of file distribution tasks generated at a plurality of moments in time; the determining the target task execution state information of the target file distribution task comprises the following steps:
according to the task execution path information of each of the plurality of file distribution tasks, distributing the file distribution tasks with the same task execution path to the same file distribution task group; and
and determining the task execution state of the file distribution tasks in the same file distribution task group according to preset rules defined for the same file distribution task group, wherein the preset rules are defined according to a mutual exclusion mechanism.
4. A method according to claim 3, wherein the preset rules comprise: the task execution status information of at most one file distribution task in the same file distribution task group indicates that the file distribution task is being executed.
5. The method of claim 3 or 4, wherein the same set of file distribution tasks includes a first file distribution task generated at a first time and a second file distribution task generated at a second time; the determining the task execution state of the file distribution task in the same file distribution task group according to the preset rule defined for the same file distribution task group comprises:
in response to determining that the first task execution state information of the first file distribution task indicates that execution is occurring and that the first time is later than the second time, the second task execution state information of the second file distribution task is updated to an ignore state characterizing skipped execution.
6. The method of claim 5, wherein determining the task execution status of the file distribution tasks in the same file distribution task group according to the preset rules defined for the same file distribution task group further comprises:
in response to determining that the first task execution state information indicates successful execution, updating the second task execution state information to successful execution; and
in response to determining that the first task execution state information indicates an execution failure, the second task execution state information is updated to the execution failure.
7. The method of claim 2, wherein the cluster further comprises an upstream node of the target node; the determining the target task execution state information of the target file distribution task comprises the following steps:
acquiring upstream task execution state information of an upstream file distribution task configured for the upstream node;
responsive to determining that the upstream task execution status information indicates that execution is not complete, re-reading the upstream task execution status information after a preset time interval; and
in response to determining that the upstream task execution state information indicates successful execution, the target task execution state information is updated to being executed.
8. The method of claim 7, wherein the determining target task execution state information for the target file distribution task further comprises:
in response to determining that the upstream task execution state information indicates an execution failure, the target task execution state information is updated to the execution failure.
9. The method of claim 2 or 7 or 8, wherein the determining target task execution state information of the target file distribution task comprises:
and in response to determining that the retry number of the target file distribution task is equal to a preset number of times, updating the execution state information of the target task to be an execution failure.
10. The method of claim 9, further comprising: before the target task execution state information is updated to an execution failure in response to determining that the number of retries of the target file distribution task is equal to a preset number of times,
and generating a second execution instruction for re-executing the target file distribution task in response to the fact that the target file distribution task fails to execute and the fact that the retry number is smaller than the preset number is determined, and adding 1 to the retry number, wherein the initial value of the retry number is 0.
11. The method of claim 9 or 10, further comprising: before the target task execution state information is updated to an execution failure in response to determining that the number of retries of the target file distribution task is equal to a preset number of times,
determining the single execution duration of the target file distribution task according to the starting time and the current time of the target file distribution task; and
and generating a third execution instruction for re-executing the target file distribution task in response to the fact that the single execution duration reaches the preset duration and the fact that the retry number is smaller than the preset number is determined, and adding 1 to the retry number.
12. The method of any of claims 1-11, further comprising: before the target node in response to determining that the cluster has obtained a file to be transferred, information of a target file distribution task configured for the target node is obtained,
responding to a file transmission instruction for representing the distribution of the files to be transmitted to the cluster, and generating the target file distribution task according to file transmission information represented by the file transmission instruction; and
and sending the information of the target file distribution task to a second external storage unit connected with each node of the cluster, so as to send the information of the target file distribution task to the node of the cluster through the second external storage unit.
13. The method of claim 11, wherein the cluster comprises a first node and a second node, the file transfer information comprising a first node identification of the first node and a second node identification of the second node; the generating the target file distribution task according to the file transmission information characterized by the file transmission instruction comprises the following steps:
and generating a target file distribution task taking the first node as a source node and the second node as a destination node according to the first node identifier and the second node identifier.
14. The method of claim 13, wherein the file transfer information further comprises a target storage address of the file to be transferred; the generating the target file distribution task according to the file transmission information characterized by the file transmission instruction further includes:
and generating a target file distribution task taking the first node as a source node, the second node as a target node and the target storage address in the second node as a file storage address of the file to be transmitted according to the first node identification, the second node identification and the target storage address.
15. The method of claim 13 or 14, wherein the sending the information of the target file distribution task to the nodes of the cluster comprises:
and according to the source node identification of the source node of the target file distribution task, the target file distribution task is sent to the node corresponding to the source node identification.
16. A file transfer device comprising:
the task information acquisition module is used for acquiring information of a target file distribution task configured for a target node in response to determining that the target node of the cluster has acquired a file to be transmitted, wherein the cluster also comprises a downstream node of the target node, and the information of the target file distribution task comprises a downstream node identifier of the downstream node; and
And the file sending module is used for responding to the received first execution instruction for executing the target file distribution task and sending the file to be transmitted to the downstream node according to the downstream node identification.
17. The apparatus of claim 16, further comprising:
the state determining module is used for determining target task execution state information of the target file distribution task in response to the fact that the current moment reaches the predefined moment; and
and the state information sending module is used for sending the target task execution state information to a first external storage unit connected with each node of the cluster, so that the target task execution state information is sent to each node of the cluster through the first external storage unit.
18. The apparatus of claim 17, wherein the target file distribution task comprises a plurality of file distribution tasks generated at a plurality of moments in time; the state determination module includes:
the task group distribution sub-module is used for distributing the file distribution tasks with the same task execution path to the same file distribution task group according to the task execution path information of each of the plurality of file distribution tasks; and
And the state determining sub-module is used for determining the task execution state of the file distribution tasks in the same file distribution task group according to preset rules defined for the same file distribution task group, wherein the preset rules are defined according to a mutual exclusion mechanism.
19. The apparatus of claim 18, wherein the preset rules comprise: the task execution status information of at most one file distribution task in the same file distribution task group indicates that the file distribution task is being executed.
20. The apparatus of claim 18 or 19, wherein the same set of file distribution tasks includes a first file distribution task generated at a first time and a second file distribution task generated at a second time; the state determination submodule includes:
and a first state updating unit configured to update second task execution state information of the second file distribution task to an ignore state characterizing skip execution in response to determining that first task execution state information of the first file distribution task indicates that the first file distribution task is being executed and that the first time is later than the second time.
21. The apparatus of claim 20, wherein the state determination submodule further comprises:
A second state updating unit configured to update the second task execution state information to execution success in response to determining that the first task execution state information indicates execution success; and
and a third state updating unit, configured to update the second task execution state information to an execution failure in response to determining that the first task execution state information indicates the execution failure.
22. The apparatus of claim 17, wherein the cluster further comprises an upstream node of the target node; the state determination module includes:
an upstream state obtaining sub-module, configured to obtain upstream task execution state information of an upstream file distribution task configured for the upstream node;
an upstream task execution status reading sub-module, configured to read the upstream task execution status information again after a preset time interval in response to determining that the upstream task execution status information indicates that the execution is not completed; and
and the first state updating sub-module is used for updating the target task execution state information into the execution state in response to determining that the upstream task execution state information indicates that the execution is successful.
23. The apparatus of claim 22, wherein the status determination module further comprises:
And the second state updating sub-module is used for updating the target task execution state information into the execution failure in response to determining that the upstream task execution state information indicates the execution failure.
24. The apparatus of claim 17 or 22 or 23, wherein the status determination module comprises:
and the third state updating sub-module is used for updating the execution state information of the target task into the execution failure in response to determining that the retry number of the target file distribution task is equal to the preset number.
25. The apparatus of claim 24, further comprising:
and the first retry sub-module is used for generating a second execution instruction for re-executing the target file distribution task in response to the fact that the target file distribution task is determined to fail to execute and the retry number is determined to be smaller than the preset number, and adding 1 to the retry number, wherein the initial value of the retry number is 0.
26. The apparatus of claim 24 or 25, further comprising:
the task duration determining submodule is used for determining the single execution duration of the target file distribution task according to the starting time and the current time of the target file distribution task; and
and the second retry sub-module is used for generating a third execution instruction for re-executing the target file distribution task and adding 1 to the retry number in response to the fact that the single execution duration reaches the preset duration and the retry number is smaller than the preset number.
27. The apparatus of any of claims 16-26, further comprising: before the target node in response to determining that the cluster has obtained a file to be transferred, information of a target file distribution task configured for the target node is obtained,
the task generating module is used for responding to the received file transmission instruction representing the file to be transmitted to the cluster, and generating the target file distribution task according to the file transmission information represented by the file transmission instruction; and
and the task information sending module is used for sending the information of the target file distribution task to a second external storage unit connected with each node of the cluster so as to send the information of the target file distribution task to the node of the cluster through the second external storage unit.
28. The apparatus of claim 26, wherein the cluster comprises a first node and a second node, the file transfer information comprising a first node identification of the first node and a second node identification of the second node; the task generating module includes:
and the first task generation sub-module is used for generating a target file distribution task taking the first node as a source node and the second node as a destination node according to the first node identification and the second node identification.
29. The apparatus of claim 28, wherein the file transfer information further comprises a target storage address of the file to be transferred; the task generating module further includes:
and the second task generating sub-module is used for generating a target file distribution task which takes the first node as a source node, the second node as a target node and the target storage address in the second node as a file storage address of the file to be transmitted according to the first node identification, the second node identification and the target storage address.
30. The apparatus of claim 28 or 29, wherein the task information transmission module comprises:
and the task information sending sub-module is used for sending the target file distribution task to a node corresponding to the source node identifier according to the source node identifier of the source node of the target file distribution task.
31. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.
32. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-15.
33. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, which, when executed by a processor, implements the method according to any one of claims 1-15.
CN202311234748.3A 2023-09-22 2023-09-22 File transmission method, device, electronic equipment and storage medium Pending CN117201483A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311234748.3A CN117201483A (en) 2023-09-22 2023-09-22 File transmission method, device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311234748.3A CN117201483A (en) 2023-09-22 2023-09-22 File transmission method, device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117201483A true CN117201483A (en) 2023-12-08

Family

ID=89005077

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311234748.3A Pending CN117201483A (en) 2023-09-22 2023-09-22 File transmission method, device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117201483A (en)

Similar Documents

Publication Publication Date Title
CN110609872A (en) Method and apparatus for synchronizing node data
CN112583931B (en) Message processing method, message middleware, electronic device, and storage medium
CN109245908B (en) Method and device for switching master cluster and slave cluster
EP3817338B1 (en) Method and apparatus for acquiring rpc member information, electronic device and storage medium
CN113742031A (en) Node state information acquisition method and device, electronic equipment and readable storage medium
CN111181765A (en) Task processing method and device
CN114448802B (en) Gateway configuration method, device, electronic equipment and storage medium
CN113641388A (en) Cloud mobile phone updating method, related device and computer program product
CN107818027B (en) Method and device for switching main name node and standby name node and distributed system
CN113849361A (en) Method, device, equipment and storage medium for testing service node
CN114070889B (en) Configuration method, traffic forwarding device, storage medium, and program product
CN111382953A (en) Dynamic process generation method and device
CN113541987A (en) Method and device for updating configuration data
CN113641688B (en) Node updating method, related device and computer program product
CN117201483A (en) File transmission method, device, electronic equipment and storage medium
CN110661857B (en) Data synchronization method and device
CN113760469A (en) Distributed computing method and device
CN113691403A (en) Topological node configuration method, related device and computer program product
CN113420275A (en) Data connection processing method, related device and computer program product
CN112860796A (en) Method, apparatus, device and storage medium for synchronizing data
CN113760487A (en) Service processing method and device
CN113360689B (en) Image retrieval system, method, related device and computer program product
CN113660339B (en) Method and apparatus for decentralizing clusters
CN112749042B (en) Application running method and device
CN113472565B (en) Method, apparatus, device and computer readable medium for expanding server function

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination