CN111881083A - File synchronization method, device, equipment and medium - Google Patents
File synchronization method, device, equipment and medium Download PDFInfo
- Publication number
- CN111881083A CN111881083A CN202010724229.5A CN202010724229A CN111881083A CN 111881083 A CN111881083 A CN 111881083A CN 202010724229 A CN202010724229 A CN 202010724229A CN 111881083 A CN111881083 A CN 111881083A
- Authority
- CN
- China
- Prior art keywords
- file
- utilization rate
- difference
- bandwidth
- transmitted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a file synchronization method, a device, equipment and a medium, comprising the following steps: monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate; determining a utilization difference between the bandwidth utilization and the CPU memory utilization; if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, performing difference comparison on the first file and the second file, and determining the difference part of the first file and the second file as a first file to be transmitted; the first file is a file of a server, and the second file is a file of a client; if the utilization rate difference is larger than a preset threshold value and the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted; and transmitting the first file to be transmitted or the second file to be transmitted to the client so that the client completes file synchronization. The file synchronization strategy can be adaptively adjusted according to the current state of the server, so that the maximization of the synchronization speed is realized by using limited resources.
Description
Technical Field
The present application relates to the field of file synchronization technologies, and in particular, to a file synchronization method, apparatus, device, and medium.
Background
In the process of uploading and downloading by a user through a cloud disk and disaster recovery backup of a storage node, data on one storage device needs to be synchronized to another device, migration types mainly comprise full volume and increment, and migration tools mainly comprise rsync and rclone. When large file increment migration is performed, if the large file is changed a little, a lot of bandwidth is wasted when the whole file is transmitted during synchronization. The existing solution is to use a difference comparison mode, such as rsync, through summary comparison and sliding block comparison, and compare the difference between two files under the condition of using a small amount of bandwidth and time complexity o (n), but the calculation amount of difference comparison is still large, and under the condition of multiple clients, the load of a server is too large, so that transmission is blocked by waiting for the calculation difference of a CPU.
Disclosure of Invention
In view of the above, an object of the present application is to provide a file synchronization method, apparatus, device and medium, which can adaptively adjust a file synchronization policy according to a current state of a server, so as to maximize a synchronization speed by using limited resources. The specific scheme is as follows:
in a first aspect, the present application discloses a file synchronization method, applied to a server, including:
monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate;
determining a utilization difference between the bandwidth utilization and the CPU memory utilization;
if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, performing difference comparison on a first file and a second file, and determining the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client;
if the utilization rate difference is larger than the preset threshold value and the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted;
and transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization.
Optionally, the monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate includes:
acquiring the bandwidth occupation amount and the CPU memory occupation amount of the current file synchronous transmission task;
determining the bandwidth utilization rate by using the bandwidth occupation amount and a preset bandwidth threshold;
and determining the CPU memory utilization rate by using the CPU memory occupation amount and the CPU memory occupation threshold value.
Optionally, the file synchronization method further includes:
and triggering the steps of acquiring the bandwidth occupation amount of the current file synchronous transmission task and the CPU memory occupation amount at regular time.
Optionally, the file synchronization method further includes:
determining the CPU memory occupation threshold according to the user level corresponding to the client and the current load state of the server;
determining the preset bandwidth threshold according to the first bandwidth threshold and the second bandwidth threshold;
the first bandwidth threshold is a bandwidth limit value allocated by the server to the client, and the second bandwidth threshold is a bandwidth limit value set by the client.
Optionally, the transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client includes:
placing the first file to be transmitted or the second file to be transmitted to a corresponding transmission queue;
and transmitting the first file to be transmitted or the second file to be transmitted in the transmission queue to the corresponding client through a socket process.
Optionally, the performing difference comparison on the first file and the second file includes:
and performing difference comparison on the first file and the second file by using a sliding block difference detection algorithm.
Optionally, the file synchronization method further includes:
comparing the time stamps of the first file and the second file;
and if the timestamps are not consistent, triggering the step of comparing the difference of the first file and the second file, and determining the difference part of the first file and the second file as a first file to be transmitted or determining the first file as a second file to be transmitted.
In a second aspect, the present application discloses a file synchronization apparatus, which is applied to a server, and includes:
the server state monitoring module is used for monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate;
a utilization difference determination module for determining a utilization difference between the bandwidth utilization and the CPU memory utilization;
the utilization rate difference judging module is used for judging whether the utilization rate difference is larger than a preset threshold value or not;
a file to be transmitted determining module, configured to compare differences between a first file and a second file if the utilization difference determining module determines that the utilization difference is greater than a preset threshold and the utilization difference determining module determines that the CPU memory utilization is less than the bandwidth utilization, and determine a difference between the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; if the utilization rate difference judging module judges that the utilization rate difference is larger than the preset threshold value and the utilization rate difference determining module determines that the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted;
and the file transmission module to be transmitted is used for transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so as to facilitate the client to complete file synchronization.
In a third aspect, the present application discloses an electronic device comprising a processor and a memory; wherein the content of the first and second substances,
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the file synchronization method.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned file synchronization method.
The method includes the steps that the bandwidth utilization rate of a current file synchronous transmission task and the CPU memory utilization rate are monitored, then a utilization rate difference value between the bandwidth utilization rate and the CPU memory utilization rate is determined, if the utilization rate difference value is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, difference comparison is conducted on a first file and a second file, and the difference portion of the first file and the second file is determined to be a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; and if the utilization rate difference is greater than the preset threshold value and the bandwidth utilization rate is less than the CPU memory utilization rate, determining the first file as a second file to be transmitted, and finally transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization. Therefore, the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate are monitored, and the file synchronization strategy can be adaptively adjusted according to the current state of the server, so that the maximization of the synchronization speed is realized by using limited resources.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a file synchronization method disclosed herein;
FIG. 2 is a flowchart of a particular file synchronization method disclosed herein;
FIG. 3 is a schematic diagram of a file synchronization apparatus disclosed in the present application;
FIG. 4 is a block diagram of an electronic device disclosed herein;
fig. 5 is a diagram of a server structure disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the process of uploading and downloading by a user through a cloud disk and disaster recovery backup of a storage node, data on one storage device needs to be synchronized to another device, migration types mainly comprise full volume and increment, and migration tools mainly comprise rsync and rclone. When large file increment migration is performed, if the large file is changed a little, a lot of bandwidth is wasted when the whole file is transmitted during synchronization. The existing solution is to use a difference comparison mode, such as rsync, through summary comparison and sliding block comparison, and compare the difference between two files under the condition of using a small amount of bandwidth and time complexity o (n), but the calculation amount of difference comparison is still large, and under the condition of multiple clients, the load of a server is too large, so that transmission is blocked by waiting for the calculation difference of a CPU. Therefore, the file synchronization scheme is provided, and the file synchronization strategy can be adaptively adjusted according to the current state of the server, so that the maximum synchronization speed is realized by using limited resources.
Referring to fig. 1, an embodiment of the present application discloses a file synchronization method, which is applied to a server, and includes:
step S11: and monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate.
In a specific implementation manner, the embodiment can acquire the bandwidth occupation amount and the CPU memory occupation amount of the current file synchronous transmission task; determining the bandwidth utilization rate by using the bandwidth occupation amount and a preset bandwidth threshold; and determining the CPU memory utilization rate by using the CPU memory occupation amount and the CPU memory occupation threshold value.
In addition, in this embodiment, the CPU memory occupation threshold may be determined according to the user level corresponding to the client and the current load state of the server; determining the preset bandwidth threshold according to the first bandwidth threshold and the second bandwidth threshold; the first bandwidth threshold is a bandwidth limit value allocated by the server to the client, and the second bandwidth threshold is a bandwidth limit value set by the client. Specifically, it may be determined that the minimum value of the first bandwidth threshold and the bandwidth threshold is the preset bandwidth threshold.
That is, in the embodiment of the present application, the cpu memory upper limit of the current file synchronous transmission task may be set according to the user level corresponding to the client and the current load state of the server; the higher the user level is, the lower the current load of the server is, and the higher the cpu memory upper limit is. And taking the minimum value as the upper limit of the bandwidth of the current transmission according to the bandwidth limit distributed to the user by the server and the bandwidth limit set by the user.
Step S12: and determining a utilization ratio difference between the bandwidth utilization ratio and the CPU memory utilization ratio.
Step S13: if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, performing difference comparison on a first file and a second file, and determining the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client.
In a specific implementation manner, the present embodiment may perform difference comparison between the first file and the second file by using a slider difference detection algorithm.
Step S14: and if the utilization rate difference is greater than the preset threshold value and the bandwidth utilization rate is less than the CPU memory utilization rate, determining the first file as a second file to be transmitted.
Step S15: and transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization.
In a specific implementation manner, the embodiment may place the first file to be transmitted or the second file to be transmitted to a corresponding transmission queue; and transmitting the first file to be transmitted or the second file to be transmitted in the transmission queue to the corresponding client through a socket process.
In addition, the embodiment can trigger the step of acquiring the bandwidth occupation amount and the CPU memory occupation amount of the current file synchronous transmission task at regular time. For example, the file synchronization policy is triggered every 10 minutes, so that the file synchronization policy can be updated regularly.
That is, the present embodiment can monitor the bandwidth utilization rate and the CPU memory utilization rate of the current file synchronous transmission task in real time. If the bandwidth occupation is close to the upper limit value and the CPU memory utilization rate is not high, the CPU memory is switched to a redundant state, more CPU memories are occupied to use a sliding block difference detection algorithm, only the difference part is transmitted to reduce the bandwidth, and therefore more files are transmitted under the limited bandwidth. If the bandwidth utilization rate is not high and the CPU memory utilization rate is close to the upper limit, the state of redundant network bandwidth is switched, the difference is not calculated any more, the files are directly compared and then transmitted, and therefore the phenomenon that the transmission queue waits for the CPU to calculate the difference and blocks transmission is avoided.
Therefore, the method and the device for synchronously transmitting the files can monitor the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate, then determine the utilization rate difference between the bandwidth utilization rate and the CPU memory utilization rate, if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, perform difference comparison on a first file and a second file, and determine the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; and if the utilization rate difference is greater than the preset threshold value and the bandwidth utilization rate is less than the CPU memory utilization rate, determining the first file as a second file to be transmitted, and finally transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization. Therefore, the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate are monitored, and the file synchronization strategy can be adaptively adjusted according to the current state of the server, so that the maximization of the synchronization speed is realized by using limited resources.
Referring to fig. 2, an embodiment of the present application discloses a specific file synchronization method, which is applied to a server and includes:
step S21: and monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate.
Step S22: and determining a utilization ratio difference between the bandwidth utilization ratio and the CPU memory utilization ratio.
Step S23: and comparing the time stamps of the first file and the second file.
Wherein, there is no order restriction between the steps S21 and S22 and the step S23.
Step S24: if the timestamps are inconsistent, the utilization ratio difference value is larger than a preset threshold value, and the CPU memory utilization ratio is smaller than the bandwidth utilization ratio, performing difference comparison on a first file and a second file, and determining the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client.
Step S25: and if the timestamps are inconsistent, the utilization rate difference value is greater than the preset threshold value, and the bandwidth utilization rate is less than the CPU memory utilization rate, determining the first file as a second file to be transmitted.
Step S26: and transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization.
That is, in this embodiment, the files of the server and the client may be compared one by one, and for each file synchronization, it may be determined whether the file is changed according to the mtime timestamp of the file, and if the server and the client both have the file and the mtime is the same, the file is not changed, and the process is ended; if the difference is not the same, the whole file is put into a transmission queue under the state that the network bandwidth is redundant, the file is compared in difference at first under the state that the CPU memory resource is redundant, then the part with the difference of the file is put into the transmission queue, the Socket process sends the content in the transmission queue to the client, and the client completes synchronization.
Referring to fig. 3, an embodiment of the present application discloses a file synchronization apparatus, which is applied to a server, and includes:
the server state monitoring module 11 is used for monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate;
a utilization difference determining module 12, configured to determine a utilization difference between the bandwidth utilization and the CPU memory utilization;
a utilization difference determining module 13, configured to determine whether the utilization difference is greater than a preset threshold;
a to-be-transmitted file determining module 14, configured to perform difference comparison on a first file and a second file if the utilization difference determining module determines that the utilization difference is greater than a preset threshold and the utilization difference determining module determines that the CPU memory utilization is less than the bandwidth utilization, and determine a difference portion between the first file and the second file as a first to-be-transmitted file; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; if the utilization rate difference judging module judges that the utilization rate difference is larger than the preset threshold value and the utilization rate difference determining module determines that the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted;
and the file transmission module 15 is configured to transmit the first file to be transmitted or the second file to be transmitted to the corresponding client, so that the client completes file synchronization.
Therefore, the method and the device for synchronously transmitting the files can monitor the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate, then determine the utilization rate difference between the bandwidth utilization rate and the CPU memory utilization rate, if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, perform difference comparison on a first file and a second file, and determine the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; and if the utilization rate difference is greater than the preset threshold value and the bandwidth utilization rate is less than the CPU memory utilization rate, determining the first file as a second file to be transmitted, and finally transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization. Therefore, the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate are monitored, and the file synchronization strategy can be adaptively adjusted according to the current state of the server, so that the maximization of the synchronization speed is realized by using limited resources.
The server state monitoring module 11 is specifically configured to acquire a bandwidth occupation amount and a CPU memory occupation amount of a current file synchronous transmission task; determining the bandwidth utilization rate by using the bandwidth occupation amount and a preset bandwidth threshold; and determining the CPU memory utilization rate by using the CPU memory occupation amount and the CPU memory occupation threshold value.
The device also comprises a timing starting module which is used for starting the server state monitoring module 11 at a timing.
The device also comprises a CPU memory occupation threshold value determining module, which is used for determining the CPU memory occupation threshold value according to the user level corresponding to the client and the current load state of the server;
the device also comprises a preset bandwidth threshold value determining module, wherein the preset bandwidth threshold value is determined according to the first bandwidth threshold value and the second bandwidth threshold value; the first bandwidth threshold is a bandwidth limit value allocated by the server to the client, and the second bandwidth threshold is a bandwidth limit value set by the client.
The file transmission module 15 to be transmitted is specifically configured to place the first file to be transmitted or the second file to be transmitted to a corresponding transmission queue; and transmitting the first file to be transmitted or the second file to be transmitted in the transmission queue to the corresponding client through a socket process.
The file to be transmitted determining module 14 is specifically configured to perform difference comparison on the first file and the second file by using a slider difference detection algorithm.
The device also comprises a file timestamp comparison module used for comparing the timestamps of the first file and the second file. Correspondingly, the file to be transmitted determining module 14 is specifically configured to, if the file timestamp comparison module determines that the timestamps are inconsistent, the utilization difference determining module determines that the utilization difference is greater than the preset threshold, and the utilization difference determining module determines that the CPU memory utilization is less than the bandwidth utilization, perform difference comparison on the first file and the second file, and determine a difference portion of the first file and the second file as the first file to be transmitted; and if the file timestamp comparison module determines that the timestamps are inconsistent, the utilization ratio difference judgment module judges that the utilization ratio difference is greater than the preset threshold value, and the utilization ratio difference determination module determines that the bandwidth utilization ratio is less than the CPU memory utilization ratio, determining the first file as a second file to be transmitted.
Referring to fig. 4, an embodiment of the present application discloses an electronic device, which includes a processor 21 and a memory 22; wherein, the memory 22 is used for saving computer programs; the processor 21 is configured to execute the computer program to implement the file synchronization method disclosed in the foregoing embodiment.
For the specific process of the file synchronization method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
Referring to fig. 5, an embodiment of the present application discloses a server 20, which includes a processor 21 and a memory 22 disclosed in the foregoing embodiments. For the steps that the processor 21 can specifically execute, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not described herein again.
Further, the server 20 in this embodiment may further specifically include a power supply 23, a communication interface 24, an input/output interface 25, and a communication bus 26; the power supply 23 is configured to provide a working voltage for each hardware device on the terminal 20; the communication interface 24 can create a data transmission channel with an external device for the terminal 20, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to a specific application requirement, which is not specifically limited herein.
Further, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the file synchronization method disclosed in the foregoing embodiment.
For the specific process of the file synchronization method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The foregoing describes a method, an apparatus, a device, and a medium for file synchronization provided in the present application in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the foregoing embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A file synchronization method is applied to a server and comprises the following steps:
monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate;
determining a utilization difference between the bandwidth utilization and the CPU memory utilization;
if the utilization rate difference is larger than a preset threshold value and the CPU memory utilization rate is smaller than the bandwidth utilization rate, performing difference comparison on a first file and a second file, and determining the difference part of the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client;
if the utilization rate difference is larger than the preset threshold value and the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted;
and transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so that the client can complete file synchronization.
2. The file synchronization method according to claim 1, wherein the monitoring of the bandwidth utilization rate and the CPU memory utilization rate of the current file synchronization transmission task comprises:
acquiring the bandwidth occupation amount and the CPU memory occupation amount of the current file synchronous transmission task;
determining the bandwidth utilization rate by using the bandwidth occupation amount and a preset bandwidth threshold;
and determining the CPU memory utilization rate by using the CPU memory occupation amount and the CPU memory occupation threshold value.
3. The file synchronization method according to claim 2, further comprising:
and triggering the steps of acquiring the bandwidth occupation amount of the current file synchronous transmission task and the CPU memory occupation amount at regular time.
4. The file synchronization method according to claim 2, further comprising:
determining the CPU memory occupation threshold according to the user level corresponding to the client and the current load state of the server;
determining the preset bandwidth threshold according to the first bandwidth threshold and the second bandwidth threshold;
the first bandwidth threshold is a bandwidth limit value allocated by the server to the client, and the second bandwidth threshold is a bandwidth limit value set by the client.
5. The file synchronization method according to claim 1, wherein the transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client comprises:
placing the first file to be transmitted or the second file to be transmitted to a corresponding transmission queue;
and transmitting the first file to be transmitted or the second file to be transmitted in the transmission queue to the corresponding client through a socket process.
6. The file synchronization method according to claim 1, wherein the performing a difference comparison between the first file and the second file comprises:
and performing difference comparison on the first file and the second file by using a sliding block difference detection algorithm.
7. The file synchronization method according to any one of claims 1 to 6, further comprising:
comparing the time stamps of the first file and the second file;
and if the timestamps are not consistent, triggering the step of comparing the difference of the first file and the second file, and determining the difference part of the first file and the second file as a first file to be transmitted or determining the first file as a second file to be transmitted.
8. A file synchronization device applied to a server includes:
the server state monitoring module is used for monitoring the bandwidth utilization rate of the current file synchronous transmission task and the CPU memory utilization rate;
a utilization difference determination module for determining a utilization difference between the bandwidth utilization and the CPU memory utilization;
the utilization rate difference judging module is used for judging whether the utilization rate difference is larger than a preset threshold value or not;
a file to be transmitted determining module, configured to compare differences between a first file and a second file if the utilization difference determining module determines that the utilization difference is greater than a preset threshold and the utilization difference determining module determines that the CPU memory utilization is less than the bandwidth utilization, and determine a difference between the first file and the second file as a first file to be transmitted; the first file is a file in a server, and the second file is a file corresponding to the first file in a client; if the utilization rate difference judging module judges that the utilization rate difference is larger than the preset threshold value and the utilization rate difference determining module determines that the bandwidth utilization rate is smaller than the CPU memory utilization rate, determining the first file as a second file to be transmitted;
and the file transmission module to be transmitted is used for transmitting the first file to be transmitted or the second file to be transmitted to the corresponding client so as to facilitate the client to complete file synchronization.
9. An electronic device comprising a processor and a memory; wherein the content of the first and second substances,
the memory is used for storing a computer program;
the processor for executing the computer program to implement the file synchronization method of any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the file synchronization method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724229.5A CN111881083A (en) | 2020-07-24 | 2020-07-24 | File synchronization method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724229.5A CN111881083A (en) | 2020-07-24 | 2020-07-24 | File synchronization method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111881083A true CN111881083A (en) | 2020-11-03 |
Family
ID=73201240
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010724229.5A Withdrawn CN111881083A (en) | 2020-07-24 | 2020-07-24 | File synchronization method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881083A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113873045A (en) * | 2021-11-11 | 2021-12-31 | 深圳市云语科技有限公司 | File multi-node transmission method based on capability negotiation |
| WO2024087914A1 (en) * | 2022-10-24 | 2024-05-02 | 超聚变数字技术有限公司 | Data synchronization method and computing device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102045197A (en) * | 2010-12-14 | 2011-05-04 | 中兴通讯股份有限公司 | Alarm data synchronization method and network management system |
-
2020
- 2020-07-24 CN CN202010724229.5A patent/CN111881083A/en not_active Withdrawn
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102045197A (en) * | 2010-12-14 | 2011-05-04 | 中兴通讯股份有限公司 | Alarm data synchronization method and network management system |
Non-Patent Citations (1)
| Title |
|---|
| 周京晖: "数据缓存按需同步的设计与应用", 《软件》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113873045A (en) * | 2021-11-11 | 2021-12-31 | 深圳市云语科技有限公司 | File multi-node transmission method based on capability negotiation |
| CN113873045B (en) * | 2021-11-11 | 2023-09-12 | 深圳市云语科技有限公司 | File multi-node transmission method based on capability negotiation |
| WO2024087914A1 (en) * | 2022-10-24 | 2024-05-02 | 超聚变数字技术有限公司 | Data synchronization method and computing device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108075934B (en) | Network quality monitoring method, device and system | |
| US9122537B2 (en) | Balancing server load according to availability of physical resources based on the detection of out-of-sequence packets | |
| US11206706B2 (en) | Method and apparatus for web browsing on multihomed mobile devices | |
| CN111881083A (en) | File synchronization method, device, equipment and medium | |
| CN105515912B (en) | Linkage availability detection and processing method between a kind of multinode | |
| CN107800574B (en) | Storage QOS adjusting method, system, equipment and computer readable memory | |
| CN110633168A (en) | Data backup method and system for distributed storage system | |
| EP4221233A1 (en) | Data download method and apparatus, computer device and storage medium | |
| CN111865674A (en) | Log processing method, device and medium | |
| CN113766013B (en) | Session creation method, device, equipment and storage medium | |
| CN111787105B (en) | File transmission method, device, computer equipment and storage medium | |
| CN111176570B (en) | Thick backup roll creating method, device, equipment and medium | |
| CN112540966A (en) | File synchronization method and device | |
| CN110753043B (en) | Communication method, device, server and medium | |
| CN113256360B (en) | Invoice processing control method, device, equipment and storage medium based on Ukey cabinet | |
| CN113518131B (en) | Fault-tolerant processing method, device and system for transmission data of network abnormality | |
| US11627630B2 (en) | TCP performance over cellular mobile networks | |
| CN110636090B (en) | Data synchronization method and device under narrow bandwidth condition | |
| CN116266283A (en) | Federal learning method and device, central server and data terminal | |
| US10177929B1 (en) | System for distributing data to multiple devices | |
| TW202215262A (en) | Reverse proxy method and storage device with delay sensing and load balancing | |
| CN110764954A (en) | Cloud site disaster tolerance method, device, equipment and storage medium | |
| CN111641698B (en) | Data statistical method, system, equipment and storage medium | |
| CN116614199B (en) | Clock synchronization method, device, slave clock equipment, clock synchronization system and medium | |
| CN109474695A (en) | A kind of data transmission method, apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201103 |
|
| WW01 | Invention patent application withdrawn after publication |