CN116756108A - File synchronization method, system, device and readable storage medium - Google Patents

File synchronization method, system, device and readable storage medium Download PDF

Info

Publication number
CN116756108A
CN116756108A CN202310807296.7A CN202310807296A CN116756108A CN 116756108 A CN116756108 A CN 116756108A CN 202310807296 A CN202310807296 A CN 202310807296A CN 116756108 A CN116756108 A CN 116756108A
Authority
CN
China
Prior art keywords
server
file
synchronization
model information
identifier
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310807296.7A
Other languages
Chinese (zh)
Inventor
夏帅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bank of China Financial Technology Co Ltd
Original Assignee
Bank of China Financial Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bank of China Financial Technology Co Ltd filed Critical Bank of China Financial Technology Co Ltd
Priority to CN202310807296.7A priority Critical patent/CN116756108A/en
Publication of CN116756108A publication Critical patent/CN116756108A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/178—Techniques for file synchronisation in file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues

Abstract

The application discloses a file synchronization method, a system, a device and a readable storage medium, which can be used in the distributed field or the financial field. The first server sends the file model information of the first server to each second server; the file model information of the first server comprises a first file identifier, wherein the first file identifier comprises file identifiers of local files of the first server, so that the second server receives the file model information of the first server and checks whether the files indicated by the first file identifiers are synchronized to obtain a synchronization check result, the synchronization check result is sent to the first server, and after the first server receives the synchronization check result sent by the second server, the first server sends an unsynchronized file in the synchronization check result to the second server. Because the first server and the second server are any two servers in the distributed system, the file synchronization of a plurality of servers in the distributed system is realized in parallel, and the file synchronization efficiency is improved.

Description

File synchronization method, system, device and readable storage medium
Technical Field
The present application relates to the field of distributed technologies, and in particular, to a method, a system, an apparatus, and a readable storage medium for file synchronization.
Background
The deployment of the distributed service is a key project of a banking system, the file synchronization in the distributed service is a heavy duty of project deployment, at present, a phenomenon that the file synchronization is not timely generally exists in the running of the distributed service, the phenomenon that after one server uploads a file, the file cannot be downloaded by another server(s) is often caused by the fact that the file synchronization is not timely, the problem belongs to serious system defects, and even the project running is caused to be abnormal, so that how to efficiently realize the file synchronization in the distributed system is a technical problem to be solved urgently at present.
Disclosure of Invention
The application provides a file synchronization method, a system, a device and a readable storage medium, which aim to improve the efficiency of file synchronization in a distributed system, and are as follows:
a file synchronization method applied to a distributed system, the distributed system comprising a plurality of servers constructed in a distributed manner, the method comprising:
the method comprises the steps that a first server obtains file model information of the first server and sends the file model information of the first server to each second server; the file model information of the first server comprises a first file identifier, the first file identifier comprises a file identifier of a local file of the first server, the first server comprises any one of the plurality of servers, and the second server comprises any one of the plurality of servers except the first server;
The second server receives the file model information of the first server and checks whether the files indicated by the first file identifications are synchronized or not to obtain a synchronization check result, wherein the synchronization check result comprises file identifications of unsynchronized files;
the second server sends the synchronization verification result to the first server;
and after receiving the synchronization verification result sent by the second server, the first server sends an unsynchronized file in the synchronization verification result to the second server.
Optionally, the first server obtains file model information of the first server, including:
the first server responds to the preset synchronization time to acquire file model information of the first server;
wherein, the reaching the preset synchronization time includes: and (3) reaching the synchronization time of a preset synchronization period, and/or detecting that the local file is updated, wherein the updating comprises deletion and/or addition.
Optionally, the first server obtains file model information of the first server, including:
the first server obtains the file name of each first file, wherein the first files comprise local files of the first server;
And the first server generates a hash value of the file name of each first file through a preset information summarization algorithm, and takes the hash value as the file identification of each first file to obtain file model information of the first server.
Optionally, the second server receives file model information of the first server, and checks whether the files indicated by the first file identifiers are synchronized, to obtain a synchronization check result, including:
the second server acquires file model information of the second server, wherein the file model information of the second server comprises a second file identifier, and the second file identifier comprises a file identifier of a local file of the second server;
the second server obtains a file identifier meeting a first preset condition as a file identifier of the unsynchronized file based on file model information of the second server and file model information of the first server, wherein the first preset condition comprises: exists in the file model information of the first server and does not exist in the file model information of the second server;
and the second server generates the synchronous check result, wherein the synchronous check result comprises file identifications of the unsynchronized files.
Optionally, the file model information of the target server further includes synchronization status information of each file identifier, where the synchronization status information includes a server identifier and a corresponding synchronization status identifier, where the corresponding synchronization status identifier of the server identifier is used to indicate whether the server indicated by the server identifier has synchronized the file indicated by the file identifier, and the target server includes a first server and a second server;
after the second server obtains the file model information of the second server, the method further includes:
the second server obtains file identifiers meeting second preset conditions as file identifiers of the difference synchronization files based on the file model information of the second server and the file model information of the first server, wherein the second preset conditions comprise: the synchronization state identification corresponding to the server identification of the second server is not synchronized in the file model information of the first server;
the synchronization verification result also comprises file identifiers of the different synchronization files.
Optionally, the first server sends the unsynchronized file in the synchronization verification result to the second server, including:
If the synchronization verification result includes the file identifier of the unsynchronized file, the first server sends the unsynchronized file to the second server, and updates the synchronization state identifier corresponding to the identifier of the second server to be synchronized in the synchronization state information of the file identifier of the unsynchronized file;
the method further comprises the steps of:
if the synchronization verification result includes the file identifier of the difference synchronization file, the first server updates the synchronization state identifier corresponding to the identifier of the second server to be synchronized in the synchronization state information of the file identifier of the difference synchronization file.
Optionally, each server in the plurality of servers respectively builds a first sending queue, a second sending queue, a first receiving queue, a second receiving queue and a file uploading queue in advance;
the first server sends file model information of the first server to each second server, and the method comprises the following steps:
the first server sends the file model information of the first server to each second server through a first sending queue;
the second server receives file model information of the first server, and the file model information comprises:
The second server receives file model information of the first server through a first receiving queue;
the second server sending a synchronization check result to the first server, comprising:
the second server sends the synchronous check result to the first server through a second sending queue;
the first server receiving the synchronization verification result sent by the second server, including:
the first server receives the synchronous verification results sent by the second servers through a second receiving queue;
the first server sending the unsynchronized file in the synchronization verification result to the second server, including:
and the first server sends the unsynchronized file in the synchronous check result to the second server through a file uploading queue.
A file synchronization system comprises a plurality of servers which are constructed in a distributed manner; wherein, the target server is used for:
acquiring file model information of the target server; the file model information of the target server comprises a target file identifier, wherein the target file identifier comprises a file identifier of a local file of the target server, and the target server comprises any one of the servers;
The file model information of the target server is sent to each other server, so that the other servers receive the file model information of the target server, check whether files indicated by the target file identifiers are synchronized or not, obtain a synchronous check result, and send the synchronous check result to the target server; wherein the other servers comprise any one server except the target server in the plurality of servers, and the synchronization verification result comprises a file identifier of an unsynchronized file;
and after receiving the synchronization verification results sent by the other servers, sending the unsynchronized files in the synchronization verification results to the other servers.
The file synchronization device is applied to a target server in a distributed system, wherein the distributed system comprises a plurality of servers which are constructed in a distributed mode, and the target server comprises any one of the servers; the device comprises:
a file information acquisition unit for acquiring file model information of the target server; the file model information of the target server comprises a target file identifier, wherein the target file identifier comprises a file identifier of a local file of the target server;
The file information sending unit is used for sending the file model information of the target server to each other server so that the other servers receive the file model information of the target server, check whether the files indicated by the target file identifications are synchronized, and generate and send a synchronization check result to the target server; wherein the other servers comprise any one server except the target server in the plurality of servers, and the synchronization verification result comprises a file identifier of an unsynchronized file;
and the file uploading unit is used for sending the unsynchronized file in the synchronous check result to the other servers after receiving the synchronous check result sent by the other servers.
A readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of a file synchronization method.
According to the file synchronization method, system, device and readable storage medium provided by the embodiment of the application, the file model information sent by the first server to each second server comprises the file identification of the local file of the first server, each second server checks whether each local file of the first server is synchronized or not according to the file identification of the local file of the second server, and feeds back the file identification of the unsynchronized file to the first server under the condition that the unsynchronized file exists, the first server can send the corresponding unsynchronized file to each second server according to the file identification of the unsynchronized file fed back by each second server, the first server comprises any one of a plurality of servers which are constructed in a distributed mode, each second server comprises any one of the plurality of servers except the first server, and therefore the application can realize that the local file in the first server is synchronized to each other server in parallel, and the file synchronization efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a file synchronization system according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of a specific implementation of a file synchronization method according to an embodiment of the present application;
FIG. 3 is a flowchart of a method for synchronizing files according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a file synchronization device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The file synchronization method provided by the embodiment of the application can be applied to a file synchronization system, as shown in fig. 1, the file synchronization system provided by the embodiment of the application comprises a plurality of servers which are constructed in a distributed manner, namely, servers 1 to n, each server respectively stores a local file, and any one server in the system, namely, a target server is used for:
and acquiring file model information of the target server, transmitting the file model information of the target server to each other server so that the other servers receive the file model information of the target server, checking whether files indicated by the target file identifiers are synchronized, and generating and transmitting a synchronization check result to the target server. And after receiving the synchronous verification results sent by the other servers, sending the unsynchronized files in the synchronous verification results to the other servers. The file model information of the target server comprises a target file identifier, the target file identifier comprises a file identifier of a local file of the target server, and the synchronous verification result comprises a file identifier of an unsynchronized file.
Therefore, each other server can check whether each local file of the target server is synchronized in the local files of the other servers according to the file identification of the local files of the target server, and feed back the file identification of the unsynchronized file to the target server when the unsynchronized file exists, and further the target server can send the corresponding unsynchronized file to each other server according to the file identification of the unsynchronized file fed back by each other server, so that the file synchronization of each server in the distributed system is realized.
Taking a distributed system including a first server and a plurality of second servers as an example, fig. 2 is a flowchart of a specific implementation method of a file synchronization method according to an embodiment of the present application, as shown in fig. 2, the method includes:
s201, the first server responds to the synchronization time reaching a preset synchronization period to acquire file model information of the first server.
In this embodiment, the synchronization period may be preconfigured, for example, the synchronization period is 24 hours, and the synchronization time of the synchronization period is 10 am, that is, the first server acquires the file model information of the first server at 10 am every day, and starts the file synchronization process.
In this embodiment, the file model information of the target servers (including the first server and each second server) in the distributed system includes a target file identifier and synchronization status information of each target file identifier, where the synchronization status information includes a server identifier and a corresponding synchronization status identifier, and the server identifier corresponds to the synchronization status identifier and is used to indicate whether the server indicated by the server identifier has synchronized the file indicated by the file identifier, and optionally, the server identifier includes a server IP address.
For example, the file model information of the first server includes a first file identifier and synchronization status information of each first file identifier, where the first file identifier includes a file identifier of a local file of the first server, and the synchronization status information of any one first file identifier includes a server identifier of each second server and a corresponding synchronization status identifier, where the synchronization status identifier includes an unsynchronized and synchronized state, and if the synchronization status identifier corresponding to the server identifier of the second server S2 is an unsynchronized state in the synchronization status information of the first file identifier A1, the unsynchronized A1 is indicated as S2.
In this embodiment, the method for the first server to obtain the file model information of the first server specifically includes:
first, the first server acquires the file names of the respective first files.
The first file includes a local file of the first server, and a specific method for obtaining a file name of the local file may refer to the prior art.
Then, the first server generates a hash value of the file name of each first file through a preset information summarization algorithm, and the hash value is used as the file identification of each first file.
In this embodiment, the message digest algorithm includes MD5 (Message Digest Algorithm MD5, fifth edition of the message digest algorithm).
And finally, the first server acquires the server identifiers of the second servers and the corresponding synchronous state identifiers of the second server identifiers through the history information.
It should be noted that, the history information includes file model information stored in the first server after the file synchronization process triggered by the previous synchronization period is finished. If the synchronization state identifier corresponding to the server identifier of the second server does not exist, the default synchronization state identifier is not synchronized.
It can be understood that the hash value of the file name generated by the MD5 algorithm is 16 bytes in size, which occupies less storage space than the file name, so that the information transmission efficiency is improved, and more file information can be sent by a single network request. The corresponding relation between the server IP address and the synchronous state identifier can be convenient for checking whether each file is completely synchronous to each server or not, and is also convenient for positioning the server corresponding to the unsynchronized file.
S202, the first server sends file model information of the first server to each second server. S203, after the second server receives the file model information of the first server, the file model information of the second server is obtained.
In this embodiment, the file model information of the second server includes the second file identifiers and synchronization status information of each second file identifier.
S204, the second server obtains file identifiers meeting the first preset conditions as file identifiers of the unsynchronized files based on the file model information of the second server and the file model information of the first server.
In this embodiment, the first preset condition includes: exists in the file model information of the first server and does not exist in the file model information of the second server.
It is understood that the file indicated by the file identifier satisfying the first preset condition is a local file existing in the first server, but a file not existing in the local file of the second server, that is, a file not synchronized to the second server.
S205, the second server obtains the file identification meeting the second preset condition as the file identification of the difference synchronization file based on the file model information of the second server and the file model information of the first server.
In this embodiment, the second preset condition includes: the synchronization state identification corresponding to the server identification of the second server is not synchronized in the file model information of the second server.
It can be understood that the file indicated by the file identifier meeting the second preset condition exists in the local files of the first server and the second server at the same time, but in the file model information of the first server, the synchronization state identifier corresponding to the server identifier of the second server is still not synchronized, that is, in the file model information of the first server, the second server defaults that the file is not synchronized yet, and obviously, the file model information of the first server is wrong and needs to be updated.
S206, the second server generates a synchronous check result and sends the synchronous check result to the first server.
In this embodiment, the synchronization verification result includes file identifiers of each unsynchronized file and file identifiers of each differentially synchronized file.
S207, the first server receives the synchronous check results sent by the second servers and judges whether file identifiers of unsynchronized files or file identifiers of different synchronous files exist in the synchronous check results. In this embodiment, the servers record different types of file identifiers according to a preconfigured type identifier rule, for example, the file identifiers of the unsynchronized files carry preset unsynchronized identifiers, the file identifiers of the differential synchronization files carry preset differential identifiers, and whether the file identifiers of the unsynchronized files or the file identifiers of the differential synchronization files exist is determined by searching the type identifiers.
Further, if the file identifier of the unsynchronized file exists, S208 is executed, if the file identifier of the differential synchronization file exists, S209 is executed, and it should be noted that, in the synchronization verification result, the file identifier of the unsynchronized file and the file identifier of the differential synchronization file may exist at the same time, so S208 and S209 may be executed in parallel or executed in series according to a preset execution rule, which is not limited in this embodiment.
S208, after the first server sends the unsynchronized file in the synchronization verification result to the second server, the synchronization state identification corresponding to the identification of the second server is updated to be synchronized in the synchronization state information of the file identification of the unsynchronized file.
It should be noted that, after the first server sends the unsynchronized file in the synchronization verification result to the second server, the synchronization status identifier corresponding to the identifier of the second server may be updated to be synchronized in the synchronization status information of the file identifier of the unsynchronized file, or may wait for the second server to complete synchronization of the unsynchronized file, and after the notification of the completion of feedback synchronization, update the synchronization status identifier corresponding to the identifier of the second server to be synchronized in the synchronization status information of the file identifier of the unsynchronized file.
S209, the first server updates the synchronous state identification corresponding to the identification of the second server to be synchronized in the synchronous state information of the file identifications of the differential synchronous files.
According to the file synchronization method provided by the embodiment of the application, the file model information sent to each second server by the first server comprises the file identification of the local file of the first server, each second server checks whether each local file of the first server is synchronized or not according to the file identification of the local file of the second server, and feeds back the file identification of the unsynchronized file to the first server under the condition that the unsynchronized file exists, and the first server can send the corresponding unsynchronized file to each second server according to the file identification of the unsynchronized file fed back by each second server, so that the local file in the first server is synchronized to each other server in parallel, and the file synchronization efficiency is improved.
Further, the file model information further comprises synchronization state information of each file identifier, on one hand, the first server can locate a server corresponding to the unsynchronized file through the synchronization state information of each file identifier and send the corresponding unsynchronized file to the server, and on the other hand, the second server can obtain the unsynchronized file and the difference synchronization file through verification of the received file model information, so that file synchronization is realized, and meanwhile, consistency of file synchronization states among the servers can be realized.
Furthermore, when the file synchronization method provided by the application is applied to each server, the method has zero invasiveness, does not relate to modifying the original codes of tasks, and overcomes the defects of system bloated, information redundancy and the like caused by the traditional third-party framework for realizing file synchronization.
It should be noted that, the file synchronization method shown in fig. 2 is only an optional specific implementation method, and the file synchronization method provided by the present application may also be implemented in other optional implementations:
for example, the synchronization time for achieving the synchronization period is only an optional synchronization opportunity, and in an optional implementation, after the first server detects that the local file is updated, file model information of the first server is obtained, where the update includes deletion and/or addition. That is, the updated file name is obtained in real time by monitoring the local file directory, and file model information is obtained.
For another example, when the synchronization time is reached, the specific method for obtaining the file model information may be to obtain the stored history file model information, and after detecting that the local file is updated, the specific method for obtaining the file model information may be to obtain the stored history file model information, and update the history file model information based on the file update condition.
For another example, in an alternative implementation, each server pre-constructs a first sending queue for sending file model information, a second sending queue for feeding back a synchronization check result, a first receiving queue for receiving file model information, a second receiving queue for receiving a synchronization check result, and a file uploading queue for sending an unsynchronized file.
Specifically, the first server sends the file model information of the first server to each second server through the first sending queue. The second server receives the file model information of the first server through the first receiving queue. And the second server sends the synchronous check result to the first server through the second sending queue. And the first server receives the synchronous check result sent by each second server through the second receiving queue. And the first server sends the unsynchronized file in the synchronous check result to the second server through the file uploading queue.
Therefore, each server realizes the transmission and the reception of file model information, the transmission and the reception of a synchronous check result and the synchronous and orderly progress of the transmission of unsynchronized files through a pre-constructed queue, and the problems of unstable system operation and slow information transmission when a large number of files are synchronized are effectively solved.
For another example, in an alternative embodiment, when the first server sends the file model information of the first server to each second server through the first sending queue, batch data in the file model information of the first server may be sent to the second server in batches through a preset sending rule, specifically, the file model information of the first server includes N pieces of target file identifiers and synchronization status information of the target file identifiers, and due to the huge number of N, the first server may send the N pieces of target file identifiers and the synchronization status information of the target file identifiers to the second server each time according to a preset time interval until all the file model information of the first server is sent to the second server.
For another example, in an alternative embodiment, the first server persists the synchronization completion information. The synchronization completion information includes synchronized information in the file model information of the first server, the synchronized information including a file identification, a server identification, and a synchronized identification. Based on this, the first server responds to the synchronization time reaching the preset synchronization period, and the acquired file model information of the first server includes file model information which is not durable.
In summary, the method for file synchronization provided by the embodiment of the present application is summarized as a flow shown in fig. 3, and as shown in fig. 3, the method specifically includes:
s301, the first server acquires file model information of the first server, and sends the file model information of the first server to each second server.
The file model information of the first server comprises a first file identifier, and the first file identifier comprises a file identifier of a local file of the first server.
S302, the second server receives file model information of the first server, and checks whether files indicated by the first file identifiers are synchronized or not to obtain a synchronization check result.
The synchronous verification result comprises file identification of the unsynchronized files.
S303, the second server sends the synchronous check result to the first server.
S304, after the first server receives the synchronous verification result sent by the second server, sending an unsynchronized file in the synchronous verification result to the second server.
According to the file synchronization method provided by the embodiment of the application, the file model information sent to each second server by the first server comprises the file identification of the local file of the first server, each second server checks whether each local file of the first server is synchronized or not according to the file identification of the local file of the second server, and feeds back the file identification of the unsynchronized file to the first server under the condition that the unsynchronized file exists, the first server can send the corresponding unsynchronized file to each second server according to the file identification of the unsynchronized file fed back by each second server, the first server comprises any server of a plurality of servers which are constructed in a distributed mode, each second server comprises any server except the first server, and therefore the method can realize that the local file in the first server is synchronized to each other server in parallel, and the file synchronization efficiency is improved.
The application also provides a file synchronization device, which can be applied to any one of a plurality of servers constructed in a distributed manner, and fig. 4 shows a schematic structure diagram of the file synchronization device, and the device comprises:
a file information obtaining unit 401, configured to obtain file model information of the target server; the file model information of the target server comprises a target file identifier, wherein the target file identifier comprises a file identifier of a local file of the target server;
a file information sending unit 402, configured to send file model information of the target server to each other server, so that the other servers receive the file model information of the target server, check whether the files indicated by the target file identifiers are synchronized, and generate and send a synchronization check result to the target server; wherein the other servers comprise any one server except the target server in the plurality of servers, and the synchronization verification result comprises a file identifier of an unsynchronized file;
and the file uploading unit 403 is configured to send an unsynchronized file in the synchronization verification result to the other server after receiving the synchronization verification result sent by the other server.
As an example, the present application further provides an application scenario and a specific implementation manner of the file synchronization method, as follows:
at present, many projects have scenes in which distributed services need to be deployed, and file services are generally necessary, but in the distributed scenes, file synchronization problems have to be considered by the file services, because after uploading files, abnormal conditions that files cannot be downloaded by another server often occur, which belongs to serious system defects, and the technical problems to be overcome specifically are as follows:
1. real-time file synchronization.
2. The integration of the three-party framework can cause project bulkiness and destroy the version management of project codes.
3. The invasiveness to the existing project code is reduced.
4. In a multi-server environment, file synchronization efficiency is low.
5. With a large number of files, the complexity of file synchronization increases.
The current common file synchronization mode comprises the step of realizing distributed file storage through an integrated third-party framework, however, the integrated third-party framework not only causes project codes to be bloated, but also has larger modification amount on the existing project codes, thereby influencing the version management of the project codes.
In order to overcome the above technical problems, the present application provides a structure of a file synchronization device specifically applied to each server, including:
The file model is used for constructing file model information, simplifying the definition of file parameter information through the construction of the file model information, specifically, the hash value is a value generated by an md5 algorithm for a file name, the size of the hash value is 16 bytes, the hash value is generally smaller than the storage space occupied by the file name, the file transmission efficiency is improved to a certain extent, more file information can be sent by a single network request, an IP set and a completion identifier are used for checking whether the file is completely synchronized to each server or not, and the hash value is also used for sending the file model information to a designated server or uploading the file in a synchronization strategy.
And the file monitoring module is used for monitoring whether the specified file directory changes, and realizing real-time synchronization of the files through the monitoring module.
And the file receiving module is used for receiving the file.
And the file uploading module is used for uploading the file.
Through the file queue, the orderly parallel processing of file synchronization and receiving is realized, and the problems of unstable and slow synchronization of a large number of files are effectively solved.
The file verification module is used for updating the file model information corresponding to the server, verifying the file synchronization completion state, persisting the success information to the local, updating the file model information corresponding to the server through the verification module, verifying the file synchronization completion state, and persisting the success information to the local.
And the file synchronization module is used for using the file queue and applying a synchronization strategy to realize parallel synchronization processing of the files. And through a file synchronization module, a multi-to-many synchronization strategy is adopted, each server simultaneously transmits local file model information to other servers in batches, a comparison result is obtained, and the comparison result is processed by a verification module and a synchronization module to realize file synchronization. If the servers are synchronized in serial order, the efficiency is greatly reduced, the servers work in parallel, the idle time is reduced, and the efficiency is improved. The synchronization module can perform initialization file synchronization after the program is started, and also has a timing synchronization strategy, so that the synchronization module can still completely synchronize under the condition of abnormal system.
All modules are mutually independent and are connected one by one, so that the full-quantity efficient synchronization of the distributed system files is ensured. Existing projects are integrated as separate modules, are non-invasive, and do not involve modification of the task's original code.
In summary, the technical effects of this scheme at least possess:
1. realizing file model definition and standardization definition: the hash value, the server IP and the completion identification set are beneficial to batch data transmission and optimize code logic processing.
2. The method realizes file queues, realizes orderly sending and receiving of files, and can well process a large number of multi-file scenes by setting queue thresholds.
3. The file synchronization strategy is realized, and a many-to-many synchronization strategy is adopted, so that data interaction is required to be performed among a plurality of servers at the same time.
4. And the file verification is used for updating the file model information corresponding to the server, verifying the file synchronization completion state and persistence-locating the successful information.
The embodiment of the application also provides a readable storage medium, which can store a computer program suitable for being executed by a processor, and when the computer program is executed by the processor, the steps of the file synchronization method provided by the embodiment of the application are realized as follows:
the method comprises the steps that a first server obtains file model information of the first server and sends the file model information of the first server to each second server; the file model information of the first server comprises a first file identifier, the first file identifier comprises a file identifier of a local file of the first server, the first server comprises any one of the plurality of servers, and the second server comprises any one of the plurality of servers except the first server;
The second server receives the file model information of the first server and checks whether the files indicated by the first file identifications are synchronized or not to obtain a synchronization check result, wherein the synchronization check result comprises file identifications of unsynchronized files;
the second server sends the synchronization verification result to the first server;
and after receiving the synchronization verification result sent by the second server, the first server sends an unsynchronized file in the synchronization verification result to the second server.
It should be noted that the method, the system, the device and the readable storage medium for file synchronization provided by the invention can be used in the distributed field or the financial field. The foregoing is merely exemplary, and the application fields of the method, the system, the device and the readable storage medium for file synchronization provided by the present invention are not limited.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A method of file synchronization, characterized by being applied to a distributed system including a plurality of servers that are distributed, the method comprising:
the method comprises the steps that a first server obtains file model information of the first server and sends the file model information of the first server to each second server; the file model information of the first server comprises a first file identifier, the first file identifier comprises a file identifier of a local file of the first server, the first server comprises any one of the plurality of servers, and the second server comprises any one of the plurality of servers except the first server;
The second server receives the file model information of the first server and checks whether the files indicated by the first file identifications are synchronized or not to obtain a synchronization check result, wherein the synchronization check result comprises file identifications of unsynchronized files;
the second server sends the synchronization verification result to the first server;
and after receiving the synchronization verification result sent by the second server, the first server sends an unsynchronized file in the synchronization verification result to the second server.
2. The method of claim 1, wherein the first server obtaining file model information for the first server comprises:
the first server responds to the preset synchronization time to acquire file model information of the first server;
wherein, the reaching the preset synchronization time includes: and (3) reaching the synchronization time of a preset synchronization period, and/or detecting that the local file is updated, wherein the updating comprises deletion and/or addition.
3. The method of claim 1, wherein the first server obtaining file model information for the first server comprises:
The first server obtains the file name of each first file, wherein the first files comprise local files of the first server;
and the first server generates a hash value of the file name of each first file through a preset information summarization algorithm, and takes the hash value as the file identification of each first file to obtain file model information of the first server.
4. The method according to claim 1, wherein the second server receives file model information of the first server, and checks whether the files indicated by the respective first file identifications have been synchronized, to obtain a synchronization check result, including:
the second server acquires file model information of the second server, wherein the file model information of the second server comprises a second file identifier, and the second file identifier comprises a file identifier of a local file of the second server;
the second server obtains a file identifier meeting a first preset condition as a file identifier of the unsynchronized file based on file model information of the second server and file model information of the first server, wherein the first preset condition comprises: exists in the file model information of the first server and does not exist in the file model information of the second server;
And the second server generates the synchronous check result, wherein the synchronous check result comprises file identifications of the unsynchronized files.
5. The method of claim 4, wherein the file model information of the target server further includes synchronization status information of each file identifier, the synchronization status information including a server identifier and a corresponding synchronization status identifier, the server identifier corresponding to the synchronization status identifier being used to indicate whether the server referred to by the server identifier has synchronized the file referred to by the file identifier, the target server including a first server and a second server;
after the second server obtains the file model information of the second server, the method further includes:
the second server obtains file identifiers meeting second preset conditions as file identifiers of the difference synchronization files based on the file model information of the second server and the file model information of the first server, wherein the second preset conditions comprise: the synchronization state identification corresponding to the server identification of the second server is not synchronized in the file model information of the first server;
The synchronization verification result also comprises file identifiers of the different synchronization files.
6. The method of claim 5, wherein the first server sending the unsynchronized file in the synchronization verification result to the second server comprises:
if the synchronization verification result includes the file identifier of the unsynchronized file, the first server sends the unsynchronized file to the second server, and updates the synchronization state identifier corresponding to the identifier of the second server to be synchronized in the synchronization state information of the file identifier of the unsynchronized file;
the method further comprises the steps of:
if the synchronization verification result includes the file identifier of the difference synchronization file, the first server updates the synchronization state identifier corresponding to the identifier of the second server to be synchronized in the synchronization state information of the file identifier of the difference synchronization file.
7. The method of claim 1, wherein each server in the plurality of servers pre-builds a first transmit queue, a second transmit queue, a first receive queue, a second receive queue, and a file upload queue;
The first server sends file model information of the first server to each second server, and the method comprises the following steps:
the first server sends the file model information of the first server to each second server through a first sending queue;
the second server receives file model information of the first server, and the file model information comprises:
the second server receives file model information of the first server through a first receiving queue;
the second server sending a synchronization check result to the first server, comprising:
the second server sends the synchronous check result to the first server through a second sending queue;
the first server receiving the synchronization verification result sent by the second server, including:
the first server receives the synchronous verification results sent by the second servers through a second receiving queue;
the first server sending the unsynchronized file in the synchronization verification result to the second server, including:
and the first server sends the unsynchronized file in the synchronous check result to the second server through a file uploading queue.
8. The file synchronization system is characterized by comprising a plurality of servers which are constructed in a distributed mode; wherein, the target server is used for:
acquiring file model information of the target server; the file model information of the target server comprises a target file identifier, wherein the target file identifier comprises a file identifier of a local file of the target server, and the target server comprises any one of the servers;
the file model information of the target server is sent to each other server, so that the other servers receive the file model information of the target server, check whether files indicated by the target file identifiers are synchronized or not, obtain a synchronous check result, and send the synchronous check result to the target server; wherein the other servers comprise any one server except the target server in the plurality of servers, and the synchronization verification result comprises a file identifier of an unsynchronized file;
and after receiving the synchronization verification results sent by the other servers, sending the unsynchronized files in the synchronization verification results to the other servers.
9. The file synchronization device is characterized by being applied to a target server in a distributed system, wherein the distributed system comprises a plurality of servers which are constructed in a distributed mode, and the target server comprises any one of the servers; the device comprises:
a file information acquisition unit for acquiring file model information of the target server; the file model information of the target server comprises a target file identifier, wherein the target file identifier comprises a file identifier of a local file of the target server;
the file information sending unit is used for sending the file model information of the target server to each other server so that the other servers receive the file model information of the target server, check whether the files indicated by the target file identifications are synchronized, and generate and send a synchronization check result to the target server; wherein the other servers comprise any one server except the target server in the plurality of servers, and the synchronization verification result comprises a file identifier of an unsynchronized file;
and the file uploading unit is used for sending the unsynchronized file in the synchronous check result to the other servers after receiving the synchronous check result sent by the other servers.
10. A readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the file synchronization method according to any one of claims 1-7.
CN202310807296.7A 2023-07-03 2023-07-03 File synchronization method, system, device and readable storage medium Pending CN116756108A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310807296.7A CN116756108A (en) 2023-07-03 2023-07-03 File synchronization method, system, device and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310807296.7A CN116756108A (en) 2023-07-03 2023-07-03 File synchronization method, system, device and readable storage medium

Publications (1)

Publication Number Publication Date
CN116756108A true CN116756108A (en) 2023-09-15

Family

ID=87947795

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310807296.7A Pending CN116756108A (en) 2023-07-03 2023-07-03 File synchronization method, system, device and readable storage medium

Country Status (1)

Country Link
CN (1) CN116756108A (en)

Similar Documents

Publication Publication Date Title
CN107332876B (en) Method and device for synchronizing block chain state
CN107465767B (en) Data synchronization method and system
CN107729366B (en) Universal multi-source heterogeneous large-scale data synchronization system
EP3562123A1 (en) Node device running method, working state switching device, node device, and medium
US7650394B2 (en) Synchronizing email recipient lists using block partition information
EP3063925B1 (en) Synchronizing event history for multiple clients
CN108563502B (en) Task scheduling method and device
CN108683668B (en) Resource checking method, device, storage medium and equipment in content distribution network
KR20080068110A (en) A method for processing data synchronization and client terminal, server and data synchronization system thereof
US20180268046A1 (en) Data processing method and apparatus
CN111259072B (en) Data synchronization method, device, electronic equipment and computer readable storage medium
KR20040111156A (en) System and method for managing cached objects using notification bonds
CN109144785B (en) Method and apparatus for backing up data
CN110046062B (en) Distributed data processing method and system
CN107231435B (en) Data synchronization monitoring method and system
US11640261B2 (en) Log processing method to avoid log collision, and related device and system
CN111831748A (en) Data synchronization method, device and storage medium
CN113704354A (en) Data synchronization method and device, computer equipment and storage medium
WO2003003656A1 (en) Method and system for auto discovery of ip-based network elements
CN111541762B (en) Data processing method, management server, device and storage medium
CN111159233B (en) Distributed caching method, system, computer equipment and storage medium
CN105323271B (en) Cloud computing system and processing method and device thereof
CN116756108A (en) File synchronization method, system, device and readable storage medium
CN111092956A (en) Resource synchronization method, device, storage medium and equipment
CN113297008B (en) Data processing method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination