CN117527789A - File download management method and system in distributed system - Google Patents
File download management method and system in distributed system Download PDFInfo
- Publication number
- CN117527789A CN117527789A CN202311638659.5A CN202311638659A CN117527789A CN 117527789 A CN117527789 A CN 117527789A CN 202311638659 A CN202311638659 A CN 202311638659A CN 117527789 A CN117527789 A CN 117527789A
- Authority
- CN
- China
- Prior art keywords
- file
- exclusive
- result
- downloading
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 24
- 238000004364 calculation method Methods 0.000 claims abstract description 31
- 239000012634 fragment Substances 0.000 claims abstract description 15
- 238000000034 method Methods 0.000 claims description 27
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012545 processing Methods 0.000 description 17
- 239000003999 initiator Substances 0.000 description 7
- 238000012790 confirmation Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention provides a file downloading management method and a file downloading management system in a distributed system, wherein a main node performs exclusive-or calculation according to a file identifier corresponding to each target fragmented file to obtain a first exclusive-or result; the working node downloads the corresponding target fragmented files according to the file identifiers, and exclusive OR calculation is carried out on the file identifiers of the target fragmented files and the file identifiers of the previous target fragmented files when one target fragmented file is downloaded; if the previous target fragment file does not exist, performing exclusive OR calculation with 0; until all the target fragmented files corresponding to the file identifications in the file downloading task are downloaded, obtaining a second exclusive-or result, and returning the downloaded target fragmented files and the second exclusive-or result to the main node; the master node judges whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, if yes, the target fragmented file is confirmed to be downloaded; the invention utilizes the exclusive-or characteristic, and the first exclusive-or result and the second exclusive-or result are the same to indicate that the file is completely downloaded.
Description
The scheme is a divisional application taking a patent application with the application date of 2023.09.12, the application number of 202311167533.4 and the name of a file download management method and system as a master application.
Technical Field
The present invention relates to the field of file transfer, and in particular, to a method and system for managing file download in a distributed system.
Background
In a download system of a distributed file system, the roles of the download system comprise a master node and a working node; file download includes three phases: the first stage: the master node receives the downloading request and acquires a file information list; and a second stage: the master node distributes corresponding file downloading tasks to the working nodes; and a third stage: after the work node finishes downloading, the work node sends confirmation information to the master node, and the master node modifies the corresponding file state from the file information list.
In summary, the summary overall flow is: the master node receiving the file downloading request needs to acquire all file information (such as file ID) and distributes the file information to the corresponding working node for downloading, and after the downloading is successful, the master node needs to feed back that a certain file is downloaded to the working node; after all the files are downloaded to the corresponding directory in the corresponding working node and confirmed to the main node, ending the downloading process corresponding to the file downloading request and feeding back the downloading result; in the process, the storage position of the file information corresponding to the coordination node is required to be memory or persistent.
The disadvantages of the above approach are: firstly, a plurality of times of IO are needed, file information is obtained once, the state is required to be modified after the completion of downloading is confirmed once, finally whether all files are downloaded is confirmed, and the file information of a main node is subjected to disc IO and performance which are wasted for a plurality of times in a database persistence mode; second, if all the file information in the master node exists in the memory in a list manner, the occupied memory capacity is too large because the downloading condition of the file is to be confirmed.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the file downloading management method and system in the distributed system are provided, and efficient confirmation of the file downloading result is achieved.
In order to solve the technical problems, the invention adopts a technical scheme that:
a file download management method in a distributed system includes the steps:
the method comprises the steps that a master node receives a file downloading request, obtains a file identifier of a target fragmented file according to the file downloading request, and sends a file downloading task to a working node according to the file identifier;
the master node performs exclusive-or calculation according to the file identification corresponding to each target fragmented file to obtain a first exclusive-or result;
the working node downloads the corresponding target fragmented file according to the file identifier, performs exclusive-or calculation on the downloaded file identifier of the target fragmented file to obtain a second exclusive-or result, and returns the downloaded target fragmented file and the second exclusive-or result to the master node;
and the master node judges whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, and if so, returns the downloaded target fragmented file to the initiator of the file downloading request.
In order to solve the technical problems, the invention adopts another technical scheme that:
a file download management system in a distributed system, comprising a master node and a working node, wherein the master node comprises a first memory, a first processor and a first computer program which is stored on the first memory and can run on the first processor, and the working node comprises a second memory, a second processor and a second computer program which is stored on the second memory and can run on the second processor;
the first processor, when executing the first computer program, performs the steps of:
receiving a file downloading request, acquiring a file identifier of a target fragmented file according to the file downloading request, and transmitting a file downloading task to a working node according to the file identifier;
performing exclusive-or calculation according to the file identification corresponding to each target fragmented file to obtain a first exclusive-or result;
judging whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, if so, returning the downloaded target fragmented file to the initiator of the file downloading request;
the second processor, when executing the second computer program, performs the steps of:
and downloading the corresponding target fragmented file according to the file identifier, performing exclusive-or calculation on the downloaded file identifier of the target fragmented file to obtain a second exclusive-or result, and returning the downloaded target fragmented file and the second exclusive-or result to a main node.
The invention has the beneficial effects that: after receiving the downloading request, the master node acquires the file identification of the target fragmented file corresponding to the file to be downloaded, and sends the file identification to the working node for downloading; the master node calculates a first exclusive or result according to the file identification of the target fragmented file; after the target fragmented file is downloaded by the working node, a second exclusive-or result is calculated according to the file identifier of the downloaded target fragmented file, and if the exclusive-or value of the first exclusive-or result and the second exclusive-or result is 0, the first exclusive-or result is identical to the second exclusive-or result, and the file downloading is completed; the judgment of the downloading condition of a plurality of target fragmented files can be realized by only storing the file identification and recording the exclusive or result of one file identification, so that the confirmation of the downloading condition is not required to be carried out through multiple times of communication between a main node and a working node, and after the main node downloads a downloading task, whether the downloading is finished can be known only through the second exclusive or result returned by the working node, thereby realizing the efficient confirmation of the downloading result of the files.
Drawings
FIG. 1 is a flowchart illustrating steps of a method for managing file downloads in a distributed system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a file download management system in a distributed system according to an embodiment of the present invention;
description of the reference numerals:
1. a master node; 11. a first processor; 12. a first memory; 2. a working node; 21. a second processor; 22. a second memory; 3. a file download management system in a distributed system.
Detailed Description
In order to describe the technical contents, the achieved objects and effects of the present invention in detail, the following description will be made with reference to the embodiments in conjunction with the accompanying drawings.
Referring to fig. 1, a method for managing file downloading in a distributed system is characterized by comprising the steps of:
the method comprises the steps that a master node receives a file downloading request, obtains a file identifier of a target fragmented file according to the file downloading request, and sends a file downloading task to a working node according to the file identifier;
the master node performs exclusive-or calculation according to the file identification corresponding to each target fragmented file to obtain a first exclusive-or result;
the working node downloads the corresponding target fragmented file according to the file identifier, performs exclusive-or calculation on the downloaded file identifier of the target fragmented file to obtain a second exclusive-or result, and returns the downloaded target fragmented file and the second exclusive-or result to the master node;
and the master node judges whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, and if so, returns the downloaded target fragmented file to the initiator of the file downloading request.
From the above description, the beneficial effects of the invention are as follows: after receiving the downloading request, the master node acquires the file identification of the target fragmented file corresponding to the file to be downloaded, and sends the file identification to the working node for downloading; the master node calculates a first exclusive or result according to the file identification of the target fragmented file; after the target fragmented file is downloaded by the working node, a second exclusive-or result is calculated according to the file identifier of the downloaded target fragmented file, and if the exclusive-or value of the first exclusive-or result and the second exclusive-or result is 0, the first exclusive-or result is identical to the second exclusive-or result, and the file downloading is completed; the judgment of the downloading condition of a plurality of target fragmented files can be realized by only storing the file identification and recording the exclusive or result of one file identification, so that the confirmation of the downloading condition is not required to be carried out through multiple times of communication between a main node and a working node, and after the main node downloads a downloading task, whether the downloading is finished can be known only through the second exclusive or result returned by the working node, thereby realizing the efficient confirmation of the downloading result of the files.
Further, the obtaining the file identifier of the target fragmented file according to the file downloading request, and sending the file downloading task to the working node according to the file identifier includes:
the master node queries information of a target file and a file identifier of a target fragment file corresponding to the target file from a file server according to the file downloading request;
and the master node judges whether the size of the target file in the information is larger than a first threshold value or whether the number of the target fragmented files is larger than a second threshold value, if so, the file identifiers are respectively formed into a plurality of file downloading tasks according to the preset number, and the file downloading tasks are sent to the working node in batches.
As can be seen from the above description, the master node queries the file server for the information of the target file, including the size of the target file and the file identifier of the target fragmented file corresponding to the target file; the target file is the file requested to be acquired by the file downloading request, the file is segmented and stored in a distributed file system in a segmented file mode, if the size of the target file exceeds a first threshold value, the target segmented file corresponding to the target file can occupy a larger memory space, under the condition that the file information is excessive, the memory overflow is possibly caused by inquiring all the file information at one time, the batched operation can be performed in batches in a paging mode and the like, the memory use condition on a main node and the memory occupation during the downloading of the file on a subsequent working node are reduced, and the batched processing can avoid the misjudgment condition caused by the fact that the data of the exclusive or is too much and is easily influenced by unexpected conditions such as network fluctuation due to the fact that the verification result is the determination of the downloading state only depends on the exclusive or result.
Further, the sending the file download task to the work node in batches includes:
and the master node respectively sends the file downloading tasks to different working nodes or sends the file downloading tasks to the same working node at intervals.
As can be seen from the above description, after the construction of the plurality of file downloading tasks is completed, the file downloading tasks can be respectively sent to different working nodes for processing and completing batch, or the plurality of file downloading tasks are sent to the same working node at intervals for completing batch; the processing process of the file can be quickened by a plurality of working nodes, the processing time is shortened by synchronously processing a plurality of file downloading tasks, and the file downloading can be traced back more quickly after the downloading is finished by one working node.
Further, the method further comprises the following steps: after receiving a file downloading request, the master node generates a task identifier corresponding to the file downloading request;
the main node performs exclusive-or calculation according to the file identifier corresponding to each target fragment file to obtain a first exclusive-or result, wherein the first exclusive-or result comprises:
each time the master node issues one file downloading task, calculating an initial exclusive OR result of file identifications in the file downloading task;
the master node judges whether a history exclusive-or result exists, if so, the initial exclusive-or result and the history exclusive-or result are subjected to exclusive-or operation to be used as a new history exclusive-or result, otherwise, the initial exclusive-or result is used as a history exclusive-or result; and obtaining the history exclusive-or result as a first exclusive-or result after all the file downloading tasks are sent, and storing the first exclusive-or result and the task identification in an associated mode.
As can be seen from the above description, each time a master node issues a file downloading task, an initial exclusive or result of a file identifier in the file downloading task is calculated; if the history exclusive-or result exists, the initial exclusive-or result and the exclusive-or value of the history exclusive-or result are calculated again to update the history exclusive-or result, if the history exclusive-or result does not exist, the initial exclusive-or result is used as the history exclusive-or result, and the characteristics of the exchange law and the combination law of the exclusive-or operation are used, so that the results obtained by exclusive-or are identical as long as the file identifications of the same batch are not influenced by the sequence, and the downloading condition of the file can be accurately identified; and the calculation is performed only when a task is issued, so that the problem of the problem under the task is prevented from influencing the comparison of exclusive OR results.
Further, the file downloading task further comprises a task identifier corresponding to the file downloading request;
the work node downloads the corresponding target fragmented file according to the file identifier, performs exclusive-or calculation on the downloaded file identifier of the target fragmented file to obtain a second exclusive-or result, and returns the downloaded target fragmented file and the second exclusive-or result to the master node, wherein the steps of:
after receiving a file downloading task, the working node downloads a corresponding target fragment file according to a file identifier in the file downloading task;
the working node performs exclusive-or calculation on the downloaded file identification of the target fragmented file to obtain a second exclusive-or result;
the working node returns the downloaded target fragment file, the task identifier and the second exclusive-or result to a master node;
the master node judging whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, if yes, returning the downloaded target fragmented file to the initiator of the file downloading request comprises:
when the master node receives a second exclusive-or result, a first exclusive-or result is obtained according to the task identifier corresponding to the second exclusive-or result;
and the master node judges whether the exclusive or value of the first exclusive or result and the final second exclusive or result is 0, and if so, returns the downloaded target fragmented file to the initiator of the file downloading request.
As can be seen from the above description, when the working node downloads according to the file downloading task, the working node performs exclusive or on the file identifier of the target fragmented file to obtain a second exclusive or result, and when the second exclusive or result is sent to the master node, the working node carries the task identifier and the downloaded target fragmented file, and because the master node may process multiple file downloading requests at the same time, the working node carries the task identifier when sending information, so that the master node can know which file downloading request corresponds to the downloading result, and the parallel processing capability of the master node on the file downloading request is improved.
Further, the master node determining whether the exclusive-or value of the first exclusive-or result and the second exclusive-or result is 0 further includes:
and if not, waiting to receive the next second exclusive-or result corresponding to the task identifier.
It can be seen from the above description that if the first exclusive-or result and the second exclusive-or result are not equal, it is indicated that the number or the value of the file identifications of the first exclusive-or result and the second exclusive-or result are different, and the master node issues the file identifications one by one in the file downloading task, and the working node downloads the file identifications according to the file downloading task, so that the file identifications of the first exclusive-or result and the second exclusive-or result are completely consistent in theory, if the first exclusive-or result and the second exclusive-or result are different, it is indicated that the query downloading task issued by the master node is not completely completed, and there is a target fragment file corresponding to the file identification to be downloaded, so that the master node continues waiting to receive and guarantees the integrity of the downloaded file.
Further, the device also comprises a status code, wherein the initial value of the status code identifies an unfinished status;
the method further comprises the following steps of:
and the master node modifies the value of the status code into a finished status, and suspends the processing task corresponding to the file downloading request until the downloaded target fragmented file returned by the working node and the second exclusive-or result are received.
As can be seen from the above description, the state of the processing task corresponding to the file download request in the master node is identified by the state code, and the whole processing process is not completely completed by the master node independently, and also involves the working node, and in the downloading process of the working node, the processing task corresponding to the master node needs to wait for the return value of the working node, so that the processing state of the master node is identified by the state code, the corresponding task can be suspended in the waiting process, and the resource consumption of the master node is reduced.
Further, the obtaining the file identifier of the target fragmented file according to the file downloading request includes:
acquiring a file identifier and a file address of the target fragmented file according to the file downloading request;
the step of downloading the corresponding target fragmented file by the working node according to the file identifier comprises the following steps:
and the working node downloads the corresponding target fragmented file according to the file address, compares the file identification of the downloaded target fragmented file with the file identification in the file downloading request, and stores the downloaded target fragmented file if the file identifications are consistent with the file identification in the file downloading request.
When the file identification of the target fragmented file is acquired, the corresponding file address can be acquired, whether the file identification of the downloaded target fragmented file is identical to the received file identification is compared after the target fragmented file is downloaded according to the file address, if so, the downloading is correct, the working node does not need to acquire the file address additionally in the downloading stage, only the file identification needs to be checked, the processing efficiency is further improved, and multiple accesses to the file server in a single task are avoided.
Further, the file is identified as a binary number.
According to the description, the file identifications are recorded in a binary number mode, the number of bits of the binary numbers is more, the problem that the same result is obtained among different file identifications when the exclusive-or value of the file identifications is calculated due to the fact that the number of bits is too small can be solved with high probability when the exclusive-or calculation is carried out, the problem can be further solved by matching with the number of the file identifications, and the accuracy of the determined downloading result is guaranteed.
Referring to fig. 2, a file download management system in a distributed system includes a master node including a first memory, a first processor, and a first computer program stored on the first memory and executable on the first processor, and a working node including a second memory, a second processor, and a second computer program stored on the second memory and executable on the second processor; the first processor executes the first computer program to realize the steps executed by the master node in the file download management method in the distributed system; and when the second processor executes the second computer program, the step executed by the working node in the file download management method in the distributed system is realized.
The file downloading management method and the file downloading management system in the distributed system can be suitable for confirming the file downloading state in the file downloading process, and particularly for confirming the file downloading state under the condition that one main node corresponds to a plurality of working nodes in the distributed file system. The following description will be made with reference to specific embodiments.
Referring to fig. 1, a first embodiment of the present invention is as follows:
a file download management method in a distributed system specifically comprises the following steps:
s1, a master node receives a file downloading request, obtains a file identifier of a target fragmented file according to the file downloading request, and sends a file downloading task to a working node according to the file identifier, wherein the method comprises the following steps:
s11, the master node queries information of a target file and a file identifier of a target fragmented file corresponding to the target file from a file server according to the file downloading request;
in an alternative embodiment, after receiving a file download request, a master node generates a task identifier (TXn) corresponding to the file download request;
in an alternative embodiment, the file identifier and the file address of the target fragmented file are obtained according to the file downloading request;
in an alternative embodiment, the file identification is a binary number;
s12, the master node judges whether the size of the target file in the information is larger than a first threshold value or whether the number of the target fragmented files is larger than a second threshold value, if yes, the file identifiers are respectively formed into a plurality of file downloading tasks according to the preset number, and the file downloading tasks are sent to a working node in batches;
wherein sending the file download task to the working node in batches comprises: the master node respectively sends a plurality of file downloading tasks to different working nodes or sends a plurality of file downloading tasks to the same working node at intervals;
s13, after the file downloading task is sent to the working node according to the file identification, the master node modifies the value of the state code into a finished state, and suspends the processing task corresponding to the file downloading request until the downloaded target fragmented file returned by the working node and the second exclusive or result are received;
in an alternative embodiment, steps S12-S13 may not be batched, which may increase the processing efficiency when the memory capacity is large and the processing resources are rich;
in an alternative embodiment, the method further comprises a download status identification code (Rn), wherein the download status identification code (Rn) comprises a task identification (TXn), a status code (F) and a first exclusive or result (BRn); the status code may be set as required, for example, in this embodiment, the status code only needs to identify the downloading condition of the file downloading task, so the status code may be set as one bit, and states that are not completely issued and that are completed after being completely issued are respectively identified through 0 and 1;
s2, the master node performs exclusive OR calculation according to the file identification corresponding to each target fragment file to obtain a first exclusive OR result (BRn), and the method comprises the following steps:
s21, calculating an initial exclusive OR result of file identifications in the file downloading task every time the master node issues one file downloading task;
in an alternative implementation manner, each time a master node sends a file downloading task, all file identifiers in the file downloading task are obtained and are subjected to exclusive-or one by one to obtain an initial exclusive-or result;
s22, the master node judges whether a history exclusive-OR result exists, if so, the initial exclusive-OR result and the history exclusive-OR result are subjected to exclusive-OR operation to be used as a new history exclusive-OR result, otherwise, the initial exclusive-OR result is used as a history exclusive-OR result; obtaining a history exclusive-or result as a first exclusive-or result after all the file downloading tasks are sent, and storing the first exclusive-or result and the task identification in an associated mode;
for example, the file downloading tasks B1 and B2 are issued together, when the issuing of the task B2 is completed, the corresponding initial exclusive-or result BR2 is obtained through calculation, and the historical exclusive-or result, namely the exclusive-or result BR1 corresponding to the task B1 is found to be stored, and then the task BR1 and the task BR2 are subjected to exclusive-or to obtain a first exclusive-or result;
s3, the working node downloads the corresponding target fragmented file according to the file identification, performs exclusive OR calculation on the downloaded file identification of the target fragmented file to obtain a second exclusive OR result (SRn), and returns the downloaded target fragmented file and the second exclusive OR result to the main node, wherein the method comprises the following steps:
s31, after receiving a file downloading task, the working node downloads a corresponding target fragment file according to a file identifier in the file downloading task;
in an optional implementation manner, the working node downloads a corresponding target fragmented file according to the file address, compares the file identifier of the downloaded target fragmented file with the file identifier in the file downloading request, and stores the downloaded target fragmented file if the file identifiers are consistent;
s32, the working node performs exclusive OR calculation on the downloaded file identification of the target fragmented file to obtain a second exclusive OR result;
in an alternative implementation manner, each time a target fragmented file is downloaded by a working node, performing exclusive or calculation on a file identifier of the target fragmented file and a file identifier of a previous target fragmented file; if the previous target fragment file does not exist, performing exclusive OR calculation with 0; until all the target fragmented files corresponding to all the file identifiers in the file downloading task are downloaded;
s33, the working node returns the downloaded target fragment file, the task identifier and the second exclusive-or result to the master node
S4, the master node judges whether the exclusive OR value of the first exclusive OR result and the second exclusive OR result is 0, if so, the downloaded target fragmented file is returned to the initiator of the file downloading request, and the method comprises the following steps:
s41, when the master node receives a second exclusive-or result, acquiring a first exclusive-or result according to the task identifier corresponding to the second exclusive-or result;
s42, the master node judges whether the exclusive-or value of the first exclusive-or result (BRn) and the final second exclusive-or result (SRn) is 0, if yes, the downloaded target fragmented file is returned to the initiator of the file downloading request, otherwise, the next second exclusive-or result corresponding to the task identifier is waited to be received.
Referring to fig. 2, a second embodiment of the present invention is as follows:
a file download management system 3 in a distributed system, comprising a master node 1 and a working node 2, the master node 1 comprising a first memory 12, a first processor 11 and a first computer program stored on the first memory 12 and executable on the first processor 11, the working node 2 comprising a second memory 22, a second processor 21 and a second computer program stored on the second memory 22 and executable on the second processor 21; the first processor 11 implements the steps executed by the master node 1 in a file download management method in a distributed system according to an embodiment when executing the first computer program; the second processor 21 executes the second computer program to implement the steps executed by the working node 2 in the file download management method in the distributed system according to the first embodiment.
In summary, the present invention provides a method and a terminal for managing file downloading in a distributed system, by using the characteristics of exclusive-or calculation, when a master node issues a file downloading task, a first exclusive-or result of a file identifier is calculated, a working node introduces the file downloading task and downloads a corresponding target fragmented file according to the file identifier, when the second exclusive-or result of the file identifier of the downloaded target fragmented file is also calculated, when all the target fragmented files corresponding to the file identifier in the downloading task are downloaded, the downloaded target fragmented file and the second exclusive-or value are returned to the master node, if the first exclusive-or result and the second exclusive-or result are 0, the master node calculates the exclusive-or value of the first exclusive-or result and the second exclusive-or result, if the first exclusive-or result is not 0, the target fragmented file corresponding to the file downloading request is not completely downloaded, and continues waiting for the second exclusive-or result uploaded by the working node; in addition, the processing capacity of the main node is considered to be capable of processing a plurality of file downloading requests simultaneously, so that corresponding task identifications are created after the file downloading requests are received, and the first exclusive-OR result and the second exclusive-OR result are both provided with the corresponding task identifications, so that the main node can clearly know the execution condition of the file downloading task corresponding to each file downloading request; and the download condition is confirmed to the main node without the need of each target fragment file downloaded by the working node, but the download condition can be directly obtained through the comparison of the first exclusive-or result and the second exclusive-or result, so that the information storage amount of the files on the main node is reduced, and the main node does not need to carry out loading update whether the downloading of the single target fragment file is completed or not.
The foregoing description is only illustrative of the present invention and is not intended to limit the scope of the invention, and all equivalent changes made by the specification and drawings of the present invention, or direct or indirect application in the relevant art, are included in the scope of the present invention.
Claims (10)
1. A file download management method in a distributed system is characterized by comprising the following steps:
the master node performs exclusive-or calculation according to the file identification corresponding to each target fragmented file to obtain a first exclusive-or result;
the working node downloads the corresponding target fragmented files according to the file identifiers, and exclusive OR calculation is carried out on the file identifiers of the target fragmented files and the file identifiers of the previous target fragmented files when one target fragmented file is downloaded; if the previous target fragment file does not exist, performing exclusive OR calculation with 0; obtaining a second exclusive-or result after all target fragmented files corresponding to all file identifiers in the file downloading task are downloaded, and returning the downloaded target fragmented files and the second exclusive-or result to a main node;
and the master node judges whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, and if so, confirms that the downloading of the target fragmented file is finished.
2. The method for managing file downloading in a distributed system according to claim 1, wherein the performing, by the master node, an exclusive-or calculation according to a file identifier corresponding to each target fragmented file to obtain a first exclusive-or result includes:
and each time the master node sends a file downloading task, acquiring all file identifiers in the file downloading task and performing exclusive-or one by one to obtain an initial exclusive-or result.
3. The method for managing file downloading in a distributed system according to claim 1, wherein before performing exclusive-or calculation according to the file identifier corresponding to each target fragmented file to obtain the first exclusive-or result, the method comprises:
and the master node receives the file downloading request, acquires the file identification of the target fragmented file according to the file downloading request, and sends the file downloading task to the working node according to the file identification.
4. A method of file download management in a distributed system according to claim 3, further comprising:
and the master node establishes corresponding task identifiers after receiving the file downloading request, and the first exclusive-or result and the second exclusive-or result both carry the corresponding task identifiers.
5. The method of claim 1, further comprising a download status identification code, wherein the download status identification code comprises a task identifier, a status code, and a first exclusive-or result; wherein the status codes identify status of not fully issued and fully issued by 0 and 1, respectively.
6. A file download management system in a distributed system, comprising a master node and a working node, wherein the master node comprises a first memory, a first processor and a first computer program which is stored on the first memory and can run on the first processor, and the working node comprises a second memory, a second processor and a second computer program which is stored on the second memory and can run on the second processor; wherein the first processor, when executing the first computer program, implements:
performing exclusive-or calculation according to the file identification corresponding to each target fragmented file to obtain a first exclusive-or result;
judging whether the exclusive or value of the first exclusive or result and the second exclusive or result is 0, if so, confirming that the downloading of the target fragmented file is finished;
the second processor is configured to:
downloading the corresponding target fragmented files according to the file identifiers, and performing exclusive OR calculation on the file identifiers of the target fragmented files and the file identifiers of the previous target fragmented files when one target fragmented file is downloaded; if the previous target fragment file does not exist, performing exclusive OR calculation with 0; and obtaining a second exclusive-or result after all the target fragmented files corresponding to the file identifications in the file downloading task are downloaded, and returning the downloaded target fragmented files and the second exclusive-or result to the main node.
7. The system for file download management in a distributed system according to claim 6, wherein performing an exclusive-or calculation according to the file identifier corresponding to each target fragmented file to obtain a first exclusive-or result comprises:
when the first processor executes the first computer program, each time a file downloading task is sent, all file identifiers in the file downloading task are obtained and are subjected to exclusive-or one by one to obtain an initial exclusive-or result.
8. The system for managing file downloads in a distributed system according to claim 6, wherein before performing the exclusive-or calculation according to the file identifier corresponding to each target fragmented file to obtain the first exclusive-or result, the method comprises:
and when the first processor executes the first computer program, receiving a file downloading request, acquiring a file identifier of a target fragment file according to the file downloading request, and sending a file downloading task to a working node according to the file identifier.
9. The file download management system in a distributed system according to claim 8, further comprising:
when the first processor executes the first computer program, a corresponding task identifier is created after a file downloading request is received, and the first exclusive-or result and the second exclusive-or result are both provided with the corresponding task identifier.
10. The system of claim 6, further comprising a download status identification code, the download status identification code comprising a task identification, a status code, and a first exclusive or result; wherein the status codes identify status of not fully issued and fully issued by 0 and 1, respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311638659.5A CN117527789A (en) | 2023-09-12 | 2023-09-12 | File download management method and system in distributed system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311167533.4A CN116915765B (en) | 2023-09-12 | 2023-09-12 | File downloading management method and system |
| CN202311638659.5A CN117527789A (en) | 2023-09-12 | 2023-09-12 | File download management method and system in distributed system |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311167533.4A Division CN116915765B (en) | 2023-09-12 | 2023-09-12 | File downloading management method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117527789A true CN117527789A (en) | 2024-02-06 |
Family
ID=88368091
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311638659.5A Pending CN117527789A (en) | 2023-09-12 | 2023-09-12 | File download management method and system in distributed system |
| CN202311167533.4A Active CN116915765B (en) | 2023-09-12 | 2023-09-12 | File downloading management method and system |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311167533.4A Active CN116915765B (en) | 2023-09-12 | 2023-09-12 | File downloading management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN117527789A (en) |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4657036B2 (en) * | 2005-07-08 | 2011-03-23 | 株式会社三共 | Game machine |
| CN101330524A (en) * | 2008-07-30 | 2008-12-24 | 华为技术有限公司 | Method and apparatus for processing download and dispatching file as well as transmission file system |
| JP5279640B2 (en) * | 2009-07-09 | 2013-09-04 | 三菱電機株式会社 | Wireless communication system, wireless communication device, receiving device, and wireless communication method |
| US10664439B2 (en) * | 2015-05-19 | 2020-05-26 | Cryptomove, Inc. | Security via dynamic data movement in a cloud-based environment |
| CN105207817B (en) * | 2015-09-21 | 2018-09-21 | 中国南方电网有限责任公司 | A kind of method that process layer devices resist network storm |
| CN106231644B (en) * | 2016-09-22 | 2017-08-25 | 国家电网公司 | The wireless sensor network data transmission method of distribution line area video monitoring |
| CN110392038B (en) * | 2019-06-03 | 2021-07-13 | 西安电子科技大学 | Multi-key searchable encryption method capable of being verified in multi-user scene |
| CN111092909A (en) * | 2019-12-30 | 2020-05-01 | 深圳市优权天成科技有限公司 | Decentralized remote control downloading algorithm and equipment |
| CN115913725A (en) * | 2022-11-21 | 2023-04-04 | 长沙学院 | Forward security dynamic searchable encryption method and system based on XOR encryption chain |
| CN115878041A (en) * | 2022-12-20 | 2023-03-31 | 苏州忆联信息系统有限公司 | Method and device for improving writing performance of RAID (redundant array of independent disks) of solid state disk and computer equipment |
-
2023
- 2023-09-12 CN CN202311638659.5A patent/CN117527789A/en active Pending
- 2023-09-12 CN CN202311167533.4A patent/CN116915765B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN116915765B (en) | 2023-12-05 |
| CN116915765A (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111897638A (en) | Distributed task scheduling method and system | |
| CN111262726B (en) | Configuration information updating method and device and computer readable storage medium | |
| CN112559558B (en) | Method and device for generating serial numbers, computing equipment and storage medium | |
| CN112765262A (en) | Data redistribution method, electronic equipment and storage medium | |
| CN110795171A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN111460504A (en) | Service processing method, device, node equipment and storage medium | |
| CN110968478A (en) | Log collection method, server and computer storage medium | |
| CN105871568B (en) | Software upgrading method and system | |
| CN106331160A (en) | Data migration method and system | |
| CN111258726A (en) | Task scheduling method and device | |
| CN110659272A (en) | Data cleaning method and system | |
| CN110417876B (en) | Session method, node server in distributed system and master control equipment | |
| CN113535262B (en) | Proxy node start control method, device, equipment and storage medium | |
| CN116915765B (en) | File downloading management method and system | |
| CN113065887A (en) | Resource processing method, resource processing device, computer equipment and storage medium | |
| CN115004666A (en) | Registration method, device, equipment and storage medium of Internet of things equipment | |
| US20220383176A1 (en) | Qubit reservation service | |
| CN114493602B (en) | Block chain transaction execution method and device, electronic equipment and storage medium | |
| CN112764897B (en) | Task request processing method, device and system and computer readable storage medium | |
| CN115277671A (en) | OTA (over the air) upgrading method and device for vehicle, vehicle and storage medium | |
| CN111343101B (en) | Server current limiting method, device, electronic equipment and readable storage medium | |
| CN113746932A (en) | Network request merging method and device, electronic device and computer program product | |
| CN116720818B (en) | Warehouse processing method, device and computer equipment based on block chain | |
| CN110737535B (en) | Data processing method and device based on message queue and computer equipment | |
| CN114780267B (en) | Interface calling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |