CN116866363A - Distributed file synchronization method, device, equipment and storage medium - Google Patents
Distributed file synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116866363A CN116866363A CN202310612879.4A CN202310612879A CN116866363A CN 116866363 A CN116866363 A CN 116866363A CN 202310612879 A CN202310612879 A CN 202310612879A CN 116866363 A CN116866363 A CN 116866363A
- Authority
- CN
- China
- Prior art keywords
- file
- synchronization
- information
- synchronized
- detail
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 230000001360 synchronised effect Effects 0.000 claims abstract description 163
- 238000012545 processing Methods 0.000 claims abstract description 30
- 230000005540 biological transmission Effects 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The application provides a distributed file synchronization method, a device, equipment and a storage medium, which can be used in the technical field of distribution. The method comprises the following steps: acquiring file information to be synchronized of a subsystem server to be synchronized in a distributed system; determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted; generating a corresponding file synchronization detail table according to the file detail information; synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table; and ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization. The distributed file synchronization method can improve the file synchronization accuracy.
Description
Technical Field
The present application relates to the field of distributed technologies, and in particular, to a method, an apparatus, a device, and a storage medium for synchronizing distributed files.
Background
In the financial field, such as banks, in the running process of a distributed system, there are scenes of file communication among different subsystems, such as transaction information synchronization and state update among the subsystems, and a general source system updates source system data according to information in a file after reading the file from a specified directory in an FTP (English is called File Transfer Protocol, chinese is file transfer protocol) mode.
At present, in the process of synchronizing and updating the files of the subsystems, partial files among the systems are possibly not synchronized, and the partial files among the systems are synchronized for a plurality of times, so that file synchronization errors are caused.
Disclosure of Invention
The application provides a distributed file synchronization method, a device, equipment and a storage medium, which are used for solving the problem that partial files among systems are not synchronized and the partial files among systems are synchronized for a plurality of times in the process of synchronizing and updating files of a subsystem at present, so that file synchronization errors are caused.
The first aspect of the present application provides a distributed file synchronization method, including:
acquiring file information to be synchronized of a subsystem server to be synchronized in a distributed system;
determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted;
generating a corresponding file synchronization detail table according to the file detail information;
synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table;
and ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
A second aspect of the present application provides a distributed file synchronization apparatus, including:
the acquisition module is used for acquiring file information to be synchronized of a subsystem server to be synchronized in the distributed system;
the determining module is used for determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted;
the generation module is used for generating a corresponding file synchronization detail table according to the file detail information;
the synchronization module is used for synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table and recording the synchronization result in the file synchronization detail table;
and the ending module is used for ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
A third aspect of the present application provides an electronic apparatus, comprising: a memory and a processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the distributed file synchronization method of any of the first aspects.
A fourth aspect of the present application provides a computer-readable storage medium having stored therein computer-executable instructions for implementing the distributed file synchronization method of any one of the first aspects when executed by a processor.
A fifth aspect of the application provides a computer program product comprising a computer program which, when executed by a processor, implements the distributed file synchronization method of any of the first aspects.
The application provides a distributed file synchronization method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring file information to be synchronized of a subsystem server to be synchronized in a distributed system; determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted; generating a corresponding file synchronization detail table according to the file detail information; synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table; and ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization. According to the distributed file synchronization method, file detail information to be synchronously transmitted is determined from a file source server based on file information to be synchronized of a subsystem server to be synchronized. Meanwhile, a corresponding file synchronization detail table is generated based on the file detail information. And synchronizing the file detail information of the subsystem server to be synchronized one by one through the file synchronization detail table when the synchronization processing is carried out, and recording a synchronization result. And when all the synchronous results corresponding to the file synchronous detail table are successful in synchronization, completing the synchronous processing. In the distributed file synchronization method, file synchronization is performed according to the file synchronization detail table, and a synchronization result is recorded. When the synchronization is unsuccessful, the synchronization is performed again until all the synchronization results are successful. Therefore, the situation that part of files among the systems are not synchronized and the part of files among the systems are synchronized for a plurality of times is avoided, and the file synchronization accuracy is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
FIG. 1 is a scene diagram of a distributed file synchronization method in which embodiments of the present application may be implemented;
FIG. 2 is a flowchart illustrating a distributed file synchronization method according to the present application;
FIG. 3 is a second flow chart of the distributed file synchronization method according to the present application;
FIG. 4 is a schematic overall flow chart of a distributed file synchronization method according to the present application;
FIG. 5 is a schematic diagram of a distributed file synchronization device according to the present application;
fig. 6 is a schematic structural diagram of an electronic device provided by the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
In the technical scheme of the embodiment of the application, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order is not violated.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
It should be noted that the method, the device, the equipment and the storage medium for synchronizing the distributed files disclosed in the disclosure can be used in the field of distributed technology. But also can be used in any field except the field of distributed technology. The application fields of the distributed file synchronization method, the device, the equipment and the storage medium are not limited.
The technical scheme of the application is described in detail below by specific examples. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
For a clear understanding of the technical solutions of the present application, the prior art solutions will be described in detail first. In the financial field, when the file synchronization processing of the subsystem is directly performed by the file transfer protocol FTP mode, the following problems may exist:
1) Information synchronization loss: in the file analysis process, if abnormal interruption occurs, the situation that only part of information is synchronized by file information and part of system information is synchronized is omitted.
2) Generating abnormal data: in the process of synchronizing information between systems, there may be a new task to be processed in addition to updating the system information. As a result of file processing multiple times, there may be situations where multiple new tasks to be processed are generated. In this case, account problems such as duplicate deductions and the like may occur to the banking system.
3) Files can only be processed once: the application system can only process the file once, and the file cannot be processed again after the abnormality occurs in the processing.
Therefore, in the current process of file synchronization and updating of the subsystems, partial files among the systems may not be synchronized, and the partial files among the systems are synchronized for a plurality of times, thereby causing file synchronization errors
Therefore, aiming at the problem that in the prior art, in the process of carrying out file synchronization and updating on the subsystems, partial files among the systems are not synchronized, and the situation that the partial files among the systems are synchronized for a plurality of times, so that file synchronization errors are caused is likely to occur, the inventor finds out in research, in order to solve the problem, a corresponding file synchronization detail table can be generated based on file detail information to be synchronously transmitted, and the file synchronization detail information of a subsystem server to be synchronized is treated one by one according to the file synchronization detail table, and the synchronization result is recorded in the file synchronization detail table during synchronization. Therefore, the situation that the part files among the systems are not synchronized and the part files among the systems are synchronized for a plurality of times can be known, and the accuracy of file synchronization is improved.
Specifically, file information to be synchronized of a subsystem server to be synchronized in a distributed system is obtained. And determining file detail information to be synchronously transmitted from the file source server according to the file information to be synchronously transmitted. And generating a corresponding file synchronization detail table according to the file detail information. Synchronizing the file detail information of the subsystem server to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table. And ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
According to the distributed file synchronization method, file detail information to be synchronously transmitted is determined from a file source server based on file information to be synchronized of a subsystem server to be synchronized. Meanwhile, a corresponding file synchronization detail table is generated based on the file detail information. When the synchronization processing is carried out, synchronizing file detail information of the subsystem server to be synchronized is processed one by one through the file synchronization detail table, and a synchronization result is recorded. And when all the synchronous results corresponding to the file synchronous detail table are successful in synchronization, completing the synchronous processing. In the distributed file synchronization method, file synchronization is performed according to the file synchronization detail table, and a synchronization result is recorded. When the synchronization is unsuccessful, the synchronization is performed again until all the synchronization results are successful. Therefore, the situation that part of files among the systems are not synchronized and the part of files among the systems are synchronized for a plurality of times is avoided, and the file synchronization accuracy is improved.
The inventor proposes the technical scheme of the application based on the creative discovery.
The application scenario of the distributed file synchronization method provided by the embodiment of the application is described below. As shown in fig. 1, 1 is an electronic device, 2 is a file source server, and 3 is a subsystem server to be synchronized. The network architecture of the application scene corresponding to the distributed file synchronization method provided by the embodiment of the application comprises the following steps: an electronic device 1, a file source server 2 and a subsystem server 3 to be synchronized. The file source server 2 may be a subsystem server in a distributed system, or may be a server dedicated to synchronizing file data. The electronic device 1 may be a management server in a distributed system.
For example, when the to-be-synchronized subsystem server 3 needs to perform file synchronization, the to-be-synchronized subsystem server 3 sends to the electronic device 1 to-be-synchronized file information. The electronic equipment 1 acquires file information to be synchronized of the subsystem server 3 to be synchronized, and determines file detail information to be synchronously transmitted from the file source server 2 according to the file information to be synchronized. The electronic equipment 1 generates a corresponding file synchronization detail table according to the file detail information, synchronizes the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table, and records the synchronization result in the file synchronization detail table. If the synchronization result of some files is unsuccessful, the file is synchronized again until all the synchronization results corresponding to the file synchronization detail table are successful, and the synchronization process is ended.
Embodiments of the present application will now be described with reference to the accompanying drawings.
Fig. 2 is a schematic flow chart of a distributed file synchronization method provided by the present application, as shown in fig. 2, in this embodiment, an execution body of the embodiment of the present application is a distributed file synchronization device, and the distributed file synchronization device may be integrated in an electronic device. The distributed file synchronization method provided in this embodiment includes the following steps:
step S101, obtaining file information to be synchronized of a subsystem server to be synchronized in a distributed system.
In this embodiment, a distributed system may include multiple subsystem servers, such as an application server, a management server, and the like. The file information to be synchronized refers to the synchronous file information required by the subsystem server to be synchronized, and the synchronous file information can be large file attribute information, such as an attribute file of a certain client of a certain service, an execution result file of a certain service at a certain time, and the like.
The method of obtaining can be obtained by receiving the synchronization request sent by the subsystem server to be synchronized, or can directly detect the file to be synchronized from the subsystem server to be synchronized, thereby generating the information of the file to be synchronized.
Step S102, determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted.
In this embodiment, the file detail information refers to detail information of various files to be synchronized, including each sub-file in each large file, specific data in the sub-file, and the like. The file source server is a server comprising files which can be used for synchronization, and because the files are stored in the file source server generally according to file attribute-specific content, corresponding file detail information can be determined from the file source server directly according to the file information to be synchronized.
Step S103, a corresponding file synchronization detail table is generated according to the file detail information.
In this embodiment, a corresponding file synchronization subtask may be generated according to the file detail information, and a corresponding file synchronization detail table may be generated based on the file synchronization subtask. Thereby providing a basis for the follow-up file synchronization according to the file synchronization detail table.
Step S104, synchronizing the file detail information of the subsystem server to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table.
The file details to be synchronized are recorded in the file synchronization detail table, and omission can be avoided and synchronization accuracy can be improved by synchronizing the file detail information of the synchronous subsystem servers one by one according to the file synchronization detail table.
Meanwhile, by recording the synchronization result in the file synchronization detail table, the synchronization result of each file can be known, thereby performing resynchronization or performing synchronization of the next file based on the synchronization result.
Step S105, when all the synchronous results corresponding to the file synchronous detail table are synchronous success, the synchronous processing is ended.
In this embodiment, when all the synchronization results corresponding to the file synchronization details table are successful in synchronization, the synchronization process is ended. If abnormal interruption occurs in the synchronization process, the synchronization is continued because the synchronization result is unsuccessful or is not performed, and the synchronization can be performed again from the interrupted position until all the synchronization results corresponding to the file synchronization detail table are successful. Thereby improving the synchronization accuracy.
The embodiment of the application provides a distributed file synchronization method, which comprises the following steps: and acquiring file information to be synchronized of a subsystem server to be synchronized in the distributed system. And determining file detail information to be synchronously transmitted from the file source server according to the file information to be synchronously transmitted. And generating a corresponding file synchronization detail table according to the file detail information. Synchronizing the file detail information of the subsystem server to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table. And ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
According to the distributed file synchronization method, file detail information to be synchronously transmitted is determined from a file source server based on file information to be synchronized of a subsystem server to be synchronized. Meanwhile, a corresponding file synchronization detail table is generated based on the file detail information. When the synchronization processing is carried out, synchronizing file detail information of the subsystem server to be synchronized is processed one by one through the file synchronization detail table, and a synchronization result is recorded. And when all the synchronous results corresponding to the file synchronous detail table are successful in synchronization, completing the synchronous processing. In the distributed file synchronization method, file synchronization is performed according to the file synchronization detail table, and a synchronization result is recorded. When the synchronization is unsuccessful, the synchronization is performed again until all the synchronization results are successful. Therefore, the situation that part of files among the systems are not synchronized and the part of files among the systems are synchronized for a plurality of times is avoided, and the file synchronization accuracy is improved.
Fig. 3 is a second flow chart of the distributed file synchronization method according to the present application, as shown in fig. 3, where the distributed file synchronization method according to the present embodiment is further refined based on the distributed file synchronization method according to the previous embodiment of the present application. The distributed file synchronization method provided in this embodiment includes the following steps.
Step S201, a synchronization request of a subsystem server to be synchronized in the distributed system is received.
In this embodiment, when a synchronization request of a subsystem server to be synchronized in a distributed system is received, the synchronization request may be processed, so as to obtain information of a file to be synchronized.
Step S202, determining file information to be synchronized from the synchronization request.
In this embodiment, the synchronization request generally includes the identifier of the subsystem server to be synchronized, the network address, the information of the file to be synchronized, and other contents, so that the information of the file to be synchronized can be directly determined from the synchronization request, and the efficiency of determining the information of the file to be synchronized is improved.
Step S203, a corresponding file synchronization task list is generated according to the file information to be synchronized.
In this embodiment, a corresponding file synchronization task may be generated according to the file information to be synchronized, and a corresponding file synchronization task table may be generated based on the file synchronization task. Each file synchronization task is recorded in the file synchronization task table, and the file synchronization task corresponds to the file to be synchronized.
Optionally, in this embodiment, S203 further includes:
and determining the synchronizable subsystem server from the distributed system according to the file information to be synchronized. The synchronizable subsystem server comprises file information to be synchronized, and is in a non-busy state.
The synchronizable subsystem server is determined to be a file source server.
When the synchronization process is performed, a corresponding synchronizable subsystem server may be first determined from the distributed system. And the synchronizable subsystem server is used as a file source server for synchronization, so that the synchronization efficiency is improved.
Meanwhile, if the number of the synchronizable subsystem servers is multiple, the synchronizable subsystem servers can be further screened, for example, the synchronizable subsystem servers with highest transmission efficiency can be screened out as file source servers based on the network transmission distance and the data transmission efficiency between the synchronizable subsystem servers and the subsystem servers to be synchronized, so that the synchronization efficiency is further improved.
Optionally, in this embodiment, S203 further includes:
judging whether file detail information corresponding to the file information to be synchronized exists or not.
If it is determined that the file detail information corresponding to the file information to be synchronized exists, a step of generating a corresponding file synchronization detail table according to the file detail information is performed, and if yes, S205 may be performed.
If it is determined that the file detail information corresponding to the file information to be synchronized does not exist, a step of determining the file detail information to be synchronously transmitted from the file source server according to the file information to be synchronized is performed, and if so, S203 may be continuously performed.
Before acquiring the file detail information, whether the file detail information corresponding to the file information to be synchronized exists or not can be judged, namely whether the file detail information is acquired or not is judged, and if the file detail information is acquired, the file detail information does not need to be acquired again, so that the problem of repeatedly acquiring the file is avoided.
Step S204, determining file detail information to be synchronously transmitted from the file source server according to the file synchronization task list.
In this embodiment, the file synchronization task table records each file synchronization task, so that the file detail information to be synchronously transmitted can be directly determined from the file source server according to the file synchronization task, for example, the file detail information to be synchronously transmitted can be obtained from the file source server by adopting a common file transmission mode.
Alternatively, in this embodiment, S204 may specifically be:
and acquiring the corresponding target file to be synchronized from the file source server in a file transmission protocol mode according to the file synchronization task list.
And analyzing the target file to generate file detail information to be synchronously transmitted.
In this embodiment, the file transfer protocol manner is adopted to improve the file transfer efficiency. After the target file is acquired, the detail information in the target file can be determined by analyzing the target file, so that the file detail information to be synchronously transmitted is obtained.
Step S205, a corresponding file synchronization subtask is generated based on the file detail information.
In this embodiment, if the file is stored in a manner of file name-subfile specific data, the file synchronization sub-tasks may respectively correspond to each subfile specific data. And when the follow-up synchronization is carried out, carrying out synchronization result statistics on the specific data of each sub-file.
And S206, generating a file synchronization detail table according to the file synchronization subtask.
In this embodiment, the file synchronization detail table may include a file synchronization subtask, a file synchronization subtask synchronization result, file detail data corresponding to the file synchronization subtask, a time of synchronization of the file synchronization subtask, and the like.
Step S207, synchronizing the file detail information of the subsystem server to be synchronized one by one according to the file synchronization detail table and recording the synchronization result in the file synchronization detail table.
The implementation of step 207 is similar to that of step 104 of the previous embodiment, and will not be described in detail here.
Step S208, when all the synchronous results corresponding to the file synchronous detail table are synchronous successfully, the synchronous processing is ended.
The implementation of step 208 is similar to that of step 105 of the previous embodiment, and will not be described in detail here.
Optionally, in this embodiment, when the synchronization result corresponding to the file synchronization detail table is unsuccessful, the following processing may be further performed in this embodiment:
and re-synchronizing the file detail information to the synchronous subsystem server according to the synchronization result of each file synchronous subtask in the file synchronous detail table.
The synchronization result of the file synchronization subtasks can be successful synchronization, unsuccessful synchronization, non-synchronization and the like. If the synchronization is not performed, the synchronization can be represented by a null, the success of the synchronization is represented by a successful identifier, and the failure of the synchronization is represented by a failed identifier. Thereby facilitating the statistics of the synchronous results.
If the synchronization is unsuccessful, the synchronization can be performed again, and if the synchronization is successful, the synchronization of other file synchronization sub-tasks can be performed.
Optionally, in this embodiment, the step of re-synchronizing the file detail information by the synchronization subsystem server according to the synchronization result of each file synchronization subtask in the file synchronization detail table may specifically be:
and synchronizing the file detail information according to the synchronization subtasks which are not successfully synchronized or are not synchronized again.
And the synchronization subtasks with successful synchronization result in the file synchronization detail table are not synchronized.
In this embodiment, the application scenario may be a scenario after an abnormal interrupt occurs, and at this time, the synchronization result in the file synchronization detail table is that the synchronization is unsuccessful or the synchronization sub-task that does not perform synchronization may implement synchronization of the file through re-synchronization processing. And the re-synchronization can be performed from the interrupt synchronization sub-task, so that repeated synchronization for a plurality of times is avoided. And the synchronous subtasks with successful synchronization result in the file synchronization detail table are not subjected to synchronous processing, so that the problem of multiple times of synchronization is avoided.
Meanwhile, in order to better understand the distributed file synchronization method of the present embodiment, the following description will be further given with reference to the accompanying drawings. As shown in fig. 4, the overall flow of the distributed file synchronization method of the present embodiment may include the following steps:
a. and acquiring the task to be synchronized, thereby generating a file synchronization task table which needs to be synchronized between systems.
b. According to the task information in the file synchronization task list, whether the file to be synchronized information needs to be acquired from the target system is judged, if not, the file which has acquired the file source server is judged, and whether the file synchronization subtask is complete, namely whether the file synchronization subtask is matched with the file synchronization task list can be further judged. And c, if the file synchronization subtask is complete, performing the processing of the step c, and if the file synchronization subtask is incomplete, acquiring the file which needs to be subjected to information synchronization from the file source server in a mode of FTP at regular time. And if necessary, acquiring the files needing to be information synchronized from the file source server in a timed FTP mode according to the file synchronization task list.
c. And c, analyzing the file according to the file obtained in the step b, generating file detail information and generating a file synchronization subtask based on the file detail information.
d. For each file synchronization subtask, a file synchronization detail table may be generated from the file synchronization subtask. And judging whether each file synchronization subtask has completed synchronization based on the file synchronization detail table, and if so, not repeating the synchronization processing. If the synchronous sub-tasks are not synchronized, information synchronization is carried out according to the synchronization processing, and meanwhile, the synchronization result of each file synchronization sub-task is updated in a file synchronization detail table.
The method in the embodiment supports the synchronization of the files among the systems of the distributed system, supports the multiple processing of the files, and supports the continuous processing of the files from the abnormal nodes.
Fig. 5 is a schematic structural diagram of a distributed file synchronization device provided by the present application, as shown in fig. 5, in this embodiment, the distributed file synchronization device 300 may be disposed in an electronic device, and the distributed file synchronization device 300 includes:
the obtaining module 301 is configured to obtain file information to be synchronized of a subsystem server to be synchronized in the distributed system.
And the determining module 302 is configured to determine file detail information to be synchronously transmitted from the file source server according to the file information to be synchronized.
And the generating module 303 is configured to generate a corresponding file synchronization detail table according to the file detail information.
And the synchronization module 304 is configured to synchronize file detail information of the subsystem server to be synchronized one by one according to the file synchronization detail table, and record a synchronization result in the file synchronization detail table.
And an ending module 305, configured to end the synchronization process when all the synchronization results corresponding to the file synchronization details table are successful in synchronization.
The distributed file synchronization device provided in this embodiment may execute the technical scheme of the method embodiment shown in fig. 2, and its implementation principle and technical effects are similar to those of the method embodiment shown in fig. 2, and are not described in detail herein.
The distributed file synchronization device provided by the present application further refines the distributed file synchronization device based on the distributed file synchronization device provided in the previous embodiment, and the distributed file synchronization device 300 includes:
optionally, in this embodiment, the obtaining module 301 is specifically configured to:
and receiving a synchronization request of a subsystem server to be synchronized in the distributed system. File information to be synchronized is determined from the synchronization request.
Optionally, in this embodiment, the determining module 302 is specifically configured to:
and generating a corresponding file synchronization task list according to the file information to be synchronized. And determining file detail information to be synchronously transmitted from the file source server according to the file synchronization task list.
Optionally, in this embodiment, when determining, according to the file synchronization task table, file detail information to be synchronously transmitted from the file source server, the determining module 302 is specifically configured to:
and acquiring the corresponding target file to be synchronized from the file source server in a file transmission protocol mode according to the file synchronization task list. And analyzing the target file to generate file detail information to be synchronously transmitted.
Optionally, in this embodiment, the generating module 303 is specifically configured to:
and generating a corresponding file synchronization subtask based on the file detail information. And generating a file synchronization detail table according to the file synchronization subtask.
Optionally, in this embodiment, when the synchronization result corresponding to the file synchronization detail table is unsuccessful, the distributed file synchronization device 300 further includes:
and the resynchronization module is used for resynchronizing the file detail information of the subsystem server to be synchronized according to the synchronization result of each file synchronization subtask in the file synchronization detail table.
Optionally, in this embodiment, the resynchronization module is specifically configured to:
and synchronizing the file detail information according to the synchronization subtasks which are not successfully synchronized or are not synchronized again.
And the synchronization subtasks with successful synchronization result in the file synchronization detail table are not synchronized.
Optionally, in this embodiment, the distributed file synchronization apparatus 300 further includes:
and the server determining module is used for determining the synchronizable subsystem server from the distributed system according to the file information to be synchronized. The synchronizable subsystem server comprises file information to be synchronized, and is in a non-busy state. The synchronizable subsystem server is determined to be a file source server.
Optionally, in this embodiment, the distributed file synchronization apparatus 300 further includes:
and the judging module is used for judging whether file detail information corresponding to the file information to be synchronized exists. And if the file detail information corresponding to the file information to be synchronized exists, executing the step of generating a corresponding file synchronization detail table according to the file detail information. And if the file detail information corresponding to the file information to be synchronized does not exist, executing the step of determining the file detail information to be synchronously transmitted from the file source server according to the file information to be synchronized.
The distributed file synchronization device provided in this embodiment may execute the technical scheme of the method embodiment shown in fig. 2 to 4, and its implementation principle and technical effects are similar to those of the method embodiment shown in fig. 2 to 4, and are not described in detail herein.
According to embodiments of the present application, the present application also provides an electronic device, a computer-readable storage medium, and a computer program product.
As shown in fig. 6, fig. 6 is a schematic structural diagram of an electronic device provided by the present application. Electronic devices are intended for various forms of digital computers, such as laptops, desktops, personal digital assistants, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 6, the electronic device includes: a processor 401 and a memory 402. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device.
Memory 402 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the distributed file synchronization method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the distributed file synchronization method provided by the present application.
The memory 402 is used as a non-transitory computer readable storage medium, and may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules (e.g., the acquisition module 301, the determination module 302, the generation module 303, the synchronization module 304, and the termination module 305 shown in fig. 5) corresponding to the distributed file synchronization method in the embodiment of the present application. The processor 401 executes various functional applications of the electronic device and data processing, i.e., implements the distributed file synchronization method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 402.
Meanwhile, the present embodiment also provides a computer product, which when executed by a processor of an electronic device, enables the electronic device to perform the distributed file synchronization method of the above embodiment.
Other implementations of the examples of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of embodiments of the application following, in general, the principles of the embodiments of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the embodiments of the application pertains.
It is to be understood that the embodiments of the application are not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be made without departing from the scope thereof. The scope of embodiments of the application is limited only by the appended claims.
Claims (12)
1. A method for synchronizing distributed files, comprising:
acquiring file information to be synchronized of a subsystem server to be synchronized in a distributed system;
determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted;
generating a corresponding file synchronization detail table according to the file detail information;
synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table, and recording the synchronization result in the file synchronization detail table;
and ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
2. The method according to claim 1, wherein the obtaining file information to be synchronized of the subsystem server to be synchronized in the distributed system includes:
receiving a synchronization request of a subsystem server to be synchronized in a distributed system;
and determining the file information to be synchronized from the synchronization request.
3. The method according to claim 2, wherein determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronized comprises:
generating a corresponding file synchronization task list according to the file information to be synchronized;
and determining file detail information to be synchronously transmitted from a file source server according to the file synchronization task list.
4. A method according to claim 3, wherein said determining file details to be synchronously transmitted from a file source server according to said file synchronization task table comprises:
acquiring a corresponding target file to be synchronized from a file source server in a file transmission protocol mode according to the file synchronization task table;
and analyzing the target file to generate file detail information to be synchronously transmitted.
5. The method of claim 4, wherein generating a corresponding file synchronization detail table from the file detail information comprises:
generating a corresponding file synchronization subtask based on the file detail information;
and generating a file synchronization detail table according to the file synchronization subtask.
6. The method according to claim 5, wherein when the synchronization result corresponding to the file synchronization detail table is unsuccessful, the method further comprises:
and re-synchronizing the file detail information of the subsystem server to be synchronized according to the synchronization result of each file synchronization subtask in the file synchronization detail table.
7. The method according to claim 6, wherein the re-synchronizing the file detail information to the synchronization subsystem server according to the synchronization result of each file synchronization sub-task in the file synchronization detail table includes:
synchronizing the file detail information according to the synchronization subtasks which are unsuccessful or not synchronized in the file synchronization detail table;
and the synchronization subtasks with successful synchronization result in the file synchronization detail table are not synchronized.
8. The method according to any one of claims 1 to 7, further comprising, before determining file details to be synchronously transmitted from a file source server according to the file information to be synchronized:
determining a synchronizable subsystem server from a distributed system according to the file information to be synchronized; the synchronizable subsystem server comprises the file information to be synchronized, and is in a non-busy state;
and determining the synchronizable subsystem server as a file source server.
9. The method according to any one of claims 4 to 7, further comprising, before determining file details to be synchronously transmitted from a file source server according to the file information to be synchronized:
judging whether file detail information corresponding to the file information to be synchronized exists or not;
if the file detail information corresponding to the file information to be synchronized exists, executing the step of generating a corresponding file synchronization detail table according to the file detail information;
and if the file detail information corresponding to the file information to be synchronized does not exist, executing the step of determining the file detail information to be synchronously transmitted from the file source server according to the file information to be synchronized.
10. A distributed file synchronization apparatus, comprising:
the acquisition module is used for acquiring file information to be synchronized of a subsystem server to be synchronized in the distributed system;
the determining module is used for determining file detail information to be synchronously transmitted from a file source server according to the file information to be synchronously transmitted;
the generation module is used for generating a corresponding file synchronization detail table according to the file detail information;
the synchronization module is used for synchronizing the file detail information of the subsystem servers to be synchronized one by one according to the file synchronization detail table and recording the synchronization result in the file synchronization detail table;
and the ending module is used for ending the synchronization processing when all the synchronization results corresponding to the file synchronization detail table are successful in synchronization.
11. An electronic device, comprising: a memory and a processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the distributed file synchronization method of any one of claims 1 to 9.
12. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing the distributed file synchronization method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310612879.4A CN116866363A (en) | 2023-05-26 | 2023-05-26 | Distributed file synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310612879.4A CN116866363A (en) | 2023-05-26 | 2023-05-26 | Distributed file synchronization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116866363A true CN116866363A (en) | 2023-10-10 |
Family
ID=88224085
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310612879.4A Pending CN116866363A (en) | 2023-05-26 | 2023-05-26 | Distributed file synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116866363A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111782619A (en) * | 2020-07-28 | 2020-10-16 | 上海爱数信息技术股份有限公司 | Document increment synchronization method and device between servers and storage medium |
-
2023
- 2023-05-26 CN CN202310612879.4A patent/CN116866363A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111782619A (en) * | 2020-07-28 | 2020-10-16 | 上海爱数信息技术股份有限公司 | Document increment synchronization method and device between servers and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109492013B (en) | Data processing method, device and system applied to database cluster | |
| CN111277639B (en) | Method and device for maintaining data consistency | |
| CN109981279B (en) | Block chain system, communication method, device, equipment and medium | |
| CN110019514B (en) | Data synchronization method and device and electronic equipment | |
| WO2022174735A1 (en) | Data processing method and apparatus based on distributed storage, device, and medium | |
| CN116866363A (en) | Distributed file synchronization method, device, equipment and storage medium | |
| CN111708794B (en) | Data comparison method and device based on big data platform and computer equipment | |
| CN113064764B (en) | Method and apparatus for executing blocks in a blockchain system | |
| CN111338834B (en) | Data storage method and device | |
| US20240097919A1 (en) | Consensus trusted cluster changing method, computer device and computer-readable storage medium | |
| CN111259022A (en) | Information synchronization method, synchronization system, computer equipment and medium | |
| CN107025257B (en) | Transaction processing method and device | |
| US20180232283A1 (en) | Service Recovery Using Snapshots and Interservice Messages | |
| WO2021082925A1 (en) | Transaction processing method and apparatus | |
| CN111327680B (en) | Authentication data synchronization method, device, system, computer equipment and storage medium | |
| CN112800060A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN108011926B (en) | Message sending method, message processing method, server and system | |
| CN109063135B (en) | Database replication method and system based on multi-activity distributed architecture | |
| EP4195033A1 (en) | Method and apparatus for upgrading blockchain system, and terminal device | |
| CN114221971B (en) | Data synchronization method, device, server, storage medium and product | |
| CN114281600A (en) | Disaster recovery backup and recovery method, device, equipment and storage medium | |
| CN111966650B (en) | Operation and maintenance big data sharing data table processing method and device and storage medium | |
| CN117010889A (en) | Data processing method, device, equipment, medium and product | |
| CN113610527A (en) | Alliance chain transaction method, device, system, terminal device and storage medium | |
| CN110445848B (en) | Method and apparatus for transaction processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |