CN116561089A - Data synchronization method, device, client and computer readable storage medium - Google Patents
Data synchronization method, device, client and computer readable storage medium Download PDFInfo
- Publication number
- CN116561089A CN116561089A CN202310834977.2A CN202310834977A CN116561089A CN 116561089 A CN116561089 A CN 116561089A CN 202310834977 A CN202310834977 A CN 202310834977A CN 116561089 A CN116561089 A CN 116561089A
- Authority
- CN
- China
- Prior art keywords
- client
- target
- data
- file
- target file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012544 monitoring process Methods 0.000 claims description 41
- 238000013475 authorization Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 10
- 230000003993 interaction Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 230000001360 synchronised effect Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000004048 modification Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data synchronization method, a data synchronization device, a client and a computer readable storage medium, wherein a target client for data synchronization with the client is determined; the target client and the client are registered clients corresponding to the same target file; and carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client. The invention adopts a mode of directly synchronizing dirty data by a plurality of clients to ensure the consistency of client cache, reduces data interaction with a data server, effectively reduces the service burden of the data server and reduces the time of data synchronization, thereby improving IO read-write performance of a distributed file system.
Description
Technical Field
The present invention relates to the field of distributed file systems, and in particular, to a data synchronization method, apparatus, client, and computer readable storage medium.
Background
In a distributed file system, in order to improve file read-write performance, a client typically uses a local memory to cache frequently accessed file data. When a file is read by multiple clients, each client is granted read rights and caches the data in local memory. The client can directly access the data in the local memory during the period of having the read permission of the file.
When a certain client needs to modify the file, the client applies for the write permission of the file, and meanwhile, the read permission of the file of other clients is recalled and the local cache data is invalid. When other clients read the file again, the clients which are granted with the write authority need to apply for the read authority again, the clients which are granted with the write authority store dirty data to the data server and release the write authority, and the clients which are granted with the read authority read the file data from the data server and store the file data to the local cache. The data synchronization mode needs to ensure the consistency of client cache by reading and writing file data through the data server, and the time consumed by data synchronization is long, so that IO read-write performance of a distributed file system is affected.
Disclosure of Invention
Accordingly, the present invention is directed to a data synchronization method, apparatus, client and computer readable storage medium, which ensure the consistency of client cache by directly synchronizing dirty data by a plurality of clients, reduce data interaction with a data server, effectively reduce the service burden of the data server and reduce the data synchronization time, thereby improving the IO read/write performance of a distributed file system.
In order to achieve the above object, the technical scheme adopted by the embodiment of the invention is as follows:
in a first aspect, the present invention provides a data synchronization method, applied to a client, the method comprising:
determining a target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file;
and carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client.
In an alternative embodiment, the performing dirty data synchronization with the target client to make the client consistent with the cached data of the target client includes:
when the client opens the target file, synchronizing dirty data of the target client to a cache of the client;
and synchronizing dirty data of the client to a cache of the target client when the client modifies the target file.
In an alternative embodiment, when the client opens the target file, synchronizing dirty data of the target client to the cache of the client includes:
when the client opens the target file, sending a dirty data query request of the target file to the target client;
receiving dirty data of the target file returned by the target client based on the dirty data query request;
and storing all the received dirty data of the target file into a cache of the client.
In an alternative embodiment, when the client modifies the target file, synchronizing dirty data of the client to a cache of the target client includes:
when the client modifies the target file, the data to be written is stored as dirty data into a cache of the client;
and sending the dirty data to the target client so that the target client can store the dirty data of the client into a cache.
In an alternative embodiment, the determining the target client that performs data synchronization with the client includes:
when the client opens the target file, a registration request is sent to a monitoring server, so that the monitoring server obtains the identifiers of all registered clients corresponding to the target file according to the registration request;
determining a target client according to the identifiers of all registered clients corresponding to the target file returned by the monitoring server; and the target client is all registered clients except the client corresponding to the target file.
In an alternative embodiment, the determining the target client that performs data synchronization with the client includes:
when the client modifies the target file, sending a write permission request of the target file to a monitoring server so that the monitoring server returns authorization information according to the write permission request; the authorization information comprises the identifications of all registered clients corresponding to the target file;
determining a target client according to the identifiers of all registered clients corresponding to the target file; and the target client is all registered clients except the client corresponding to the target file.
In a second aspect, the present invention provides a data synchronization device, applied to a client, the device comprising:
the decision module is used for determining a target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file;
and the synchronization module is used for carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client.
In an alternative embodiment, the synchronization module is configured to synchronize dirty data of the target client to a cache of the client when the client opens the target file; and synchronizing dirty data of the client to a cache of the target client when the client modifies the target file.
In a third aspect, the present invention provides a client comprising a memory for storing a computer program and a processor for executing the data synchronization method according to any one of the previous embodiments when the computer program is invoked.
In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data synchronization method as in any of the previous embodiments.
Compared with the prior art, the data synchronization method, the device, the client and the computer readable storage medium provided by the embodiment of the invention determine the target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file; and carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client. The invention adopts a mode of directly synchronizing dirty data by a plurality of clients to ensure the consistency of client cache, reduces data interaction with a data server, effectively reduces the service burden of the data server and reduces the time of data synchronization, thereby improving IO read-write performance of a distributed file system.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows an application scenario schematic diagram provided by an embodiment of the present invention.
Fig. 2 shows a schematic diagram of a data synchronization method according to an embodiment of the present invention.
Fig. 3 shows a schematic diagram of the sub-steps of step S20 in fig. 2.
Fig. 4 shows a schematic diagram of the sub-steps of steps S201 and S202 in fig. 3.
Fig. 5 shows a schematic diagram of the sub-steps of step S10 in fig. 4.
Fig. 6 is a block diagram of a data synchronization device according to an embodiment of the present invention.
Fig. 7 shows a block schematic diagram of a client according to an embodiment of the present invention.
Icon: 10-client; 20-a monitoring server; 30-a data server; 110-memory; a 120-processor; 130-a communication module; 200-a data synchronization device; 201-a decision module; 202-synchronization module.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In a distributed file system, in order to improve the performance of the file system, different types of cache storage systems are often introduced, and the cache storage systems are generally divided into two types, one type is a distributed cache, such as Memcached, redis, tair and the like; the other type is a local cache, such as a cache of Facebook, a solid state disk, a local memory for storing data, and the like, wherein the solid state disk belongs to the local disk cache, and the local memory for storing data belongs to the local memory cache.
In order to improve the performance of a file system and the hit rate of hot spot data, a local memory is generally introduced into a client to cache frequently accessed file data. Once the memory cache data is introduced at the client, it is important to ensure the consistency of the caches of the clients. At present, a data server is generally used for transferring to ensure the consistency of client-side caches, namely, a client-side stores dirty data to the data server, and other client-sides acquire data from the data server to update a local cache, so that the data cached by all client-sides which open the same file are completely identical. The data is read and written for a long time by adopting a data server transferring mode, so that IO read and write performance of the distributed file system is affected.
Based on the above, the embodiments of the present invention provide a data synchronization method, apparatus, client and computer readable storage medium, which adopt a manner that a plurality of clients directly synchronize dirty data to ensure the consistency of client caches, reduce data interaction with a data server, effectively reduce the service burden of the data server and reduce the time of data synchronization, thereby improving the IO read-write performance of a distributed file system.
Referring to fig. 1, fig. 1 shows a schematic view of an application scenario suitable for an embodiment of the present invention. In fig. 1, a plurality of clients 10 are communicatively connected to each other, and each client 10 is communicatively connected to a monitoring server 20 and a data server 30, respectively. The client 10 responds to an access request of a user for a file, where the access request may include opening the file, modifying a file attribute, and additionally writing or overwriting data on the file, where the modification of the file attribute, the additional writing or overwriting data on the file are all modification operations on the file. The client 10 executes steps S10 to S20 and corresponding sub-steps in the following examples according to the received specific request to achieve the corresponding technical effects. The monitoring server 20 is mainly responsible for monitoring the files opened by the client 10 and controlling the file write rights. The data server 30 is mainly responsible for storing data and metadata generated by the client 10 and processing requests for the client 10 to read the data or metadata.
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 2 is a schematic diagram of a data synchronization method according to an embodiment of the present invention, where an execution body of the method may be one of the clients described above, and referring to fig. 2, the method includes the following steps:
step S10, determining a target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file.
In the embodiment of the invention, when the client opens the file, the client needs to register. Assuming that the client accesses the file A, the file A is the target file, and other clients which have already opened the target file can possibly read and write the target file, in order to ensure that each client can access the latest data of the target file, firstly, the registered client corresponding to the target file is acquired and used as the target client.
Step S20, carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client.
In the embodiment of the invention, the dirty data is synchronized between the client and the target client so that the data cached by the client and the target client are consistent, that is, if the client has the dirty data, the client synchronizes the dirty data to the target client so that the target client stores the received dirty data in the local cache; and if the target client has dirty data, the dirty data of the target client is synchronously saved in the local cache, so that a plurality of client caches corresponding to the same target file are ensured to be consistent.
In summary, in the data synchronization method provided by the embodiment of the present invention, first, a target client performing data synchronization with a client is determined; the target client and the client are registered clients corresponding to the same target file; and performing dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client. The invention adopts a mode of directly synchronizing dirty data by a plurality of clients to ensure the consistency of client cache, reduces data interaction with a data server, effectively reduces the service burden of the data server and reduces the time of data synchronization, thereby improving IO read-write performance of a distributed file system.
It should be noted that, the monitoring server generates or updates the management data according to the operation of the client, where the management data includes a registry for recording the opened file and a lock authority table for recording the authorization condition of the write authority of the file.
As an implementation manner, the monitoring server records the corresponding relation between the file inode number and the file client opening through a registry, and the registry records all the opened file inode numbers and all the identifiers of the clients opening the file, wherein the file inode numbers are unique integer values used for identifying the files in a file system, and the identifiers of the clients can be client names, client IDs or client IP addresses.
Suppose that a file with file inode number 10 has been opened by client B and client N, where client B has an IP address of 8.8.8.8 and client N has an IP address of 8.8.8.9. Client registration information with file inode number 10 is recorded in the registry of the monitoring server, as shown in table 1.
TABLE 1
In the embodiment of the invention, in order to improve the read-write performance of the file, the file can be divided into a plurality of specific ranges, and the range lock is used for ensuring that a plurality of clients do not modify the same specific range of the target file at the same time. For example, a file is split into pages according to a preset page size, and assuming that the page size is 64M, a 100M file is split into 2 pages, and the page codes are 0 and 1.
As one embodiment, the monitoring server records, through the lock authority table, the correspondence between the file inode number, the page code, and the client holding the file page write authority, assuming that the ID of the client B is 1, the file inode number is 10, and the range lock write authority with the page code of 0 is held by the client B. The lock authority table of the monitoring server records the write authority authorization condition of the range lock with the file inode number of 10 and the page code of 0, as shown in table 2.
TABLE 2
Optionally, in practical applications, when the client opens or modifies the target file, data synchronization with the target client is required. Referring to fig. 3, the substeps of step S20 may include:
in step S201, when the client opens the target file, dirty data of the target client is synchronized to the cache of the client.
In the embodiment of the invention, when the client responds to the request of a user to open the target file, dirty data which is not stored in the cache of the target client to the data server is acquired, and the acquired dirty data is synchronized into the cache of the client, so that the client which newly opens the target file can access the latest data.
In one embodiment, when the client synchronizes the dirty data of the target client to the local cache, the synchronization information may include a file inode number, a total number of pages of the dirty data, a page code corresponding to the dirty data, a data length of the dirty data, and the dirty data.
In step S202, when the client modifies the target file, dirty data of the client is synchronized to the cache of the target client.
In the embodiment of the invention, when the client modifies the target file through operations such as additional writing, overwriting or modifying attribute information, dirty data is correspondingly generated, and the dirty data newly generated by the client is synchronized into the cache of the target client so as to ensure that the target client can read the latest data or update the latest data.
As one implementation mode, the client side synchronizes the dirty data to the target client side in an asynchronous distribution mode, and the asynchronous synchronization information can comprise a file inode number, a page code corresponding to the dirty data, the data length of the dirty data and the dirty data.
Optionally, in practical application, the corresponding cache data of the target file newly opened by the client is constructed by acquiring the dirty data of the target file in the target client, so as to ensure that the client can access the latest data of the target file. Referring to fig. 4, the sub-steps of step S201 may include:
in step S2011, when the client opens the target file, a dirty data query request of the target file is sent to the target client.
In the embodiment of the invention, each client only caches the latest data of the opened file, and when the client closes the file, the client clears the cached data of the closed file, so that the target client may have dirty data of the target file, and when the client opens one target file, a dirty data query request needs to be sent to the target client, wherein the query request comprises the inode number of the target file.
After receiving the dirty data query request, the target client queries the local cache by using the file inode number carried in the query request, acquires corresponding dirty data, and returns the dirty data to the client.
In step S2012, the dirty data of the target file returned by the target client based on the dirty data query request is received.
Step S2013, storing all dirty data of the received target file in the cache of the client.
In the embodiment of the invention, the client sends a request for acquiring the dirty data to all the target clients, but only a part of the target clients may have the dirty data, or each target client may have the dirty data or have no dirty data, so that the dirty data of the target file is acquired according to the received return result of each target client, and finally all the dirty data of the target file are saved in the cache of the client.
Optionally, in practical application, dirty data generated by modifying the target file by the client is synchronized to all target clients, so that each target client can access the latest data of the target file. Referring to fig. 4, the sub-steps of step S202 may include:
in step S2021, when the client modifies the target file, the data to be written is saved as dirty data to the cache of the client.
Step S2022 sends the dirty data to the target client so that the target client saves the dirty data of the client to the cache.
In the embodiment of the invention, when the client modifies the target file, corresponding data to be written is generated, the data to be written comprises metadata and data of the target file, the data to be written is stored as dirty data in a local cache of the client, the dirty data is sent to the target client, and the target client stores the received dirty data in the cache so as to be capable of accessing the latest data of the target file.
It should be noted that, assuming that dirty data of the nth page of the target file already exists in the target client cache B, the client a modifies the nth page of the target file to generate dirty data, sends the dirty data to the target client B, and after receiving the dirty data synchronization request of the nth page of the target file, the target client B queries the state of the nth page of the target file in the local cache, if the state is normal, stores the dirty data in the local cache, if the state is dirty data, updates the state to normal, and stores the dirty data in the local cache. This ensures that multiple clients do not have duplicate dirty data.
Optionally, in practical application, to ensure that the client newly opening the target file can access the latest data of the target file, the target client is acquired from the monitoring server, so as to synchronize the dirty data of the target client to the local. Referring to fig. 5, the substeps of step S10 may include:
in step S101, when the client opens the target file, a registration request is sent to the monitoring server, so that the monitoring server obtains the identifiers of all registered clients corresponding to the target file according to the registration request.
In the embodiment of the invention, when the client opens the target file, the client sends a registration request to the monitoring server, wherein the registration request comprises the identification of the client A and the target file inode number, and the monitoring server updates the registry according to the received registration request.
The monitoring server sends the identification of the newly registered client to the registered client so that the registered client can synchronize the newly generated dirty data to the newly registered client subsequently, and therefore cache consistency among the clients is achieved. Meanwhile, the monitoring server distributes a uniformly addressed client ID for the first registered client so as to apply for writing permission when the client modifies the file. And finally, the monitoring server sends the ID of the newly registered client and the identification of the registered client corresponding to the target file to the newly registered client so that the newly registered client and the target client carry out dirty data synchronization.
Step S102, determining a target client according to the identifiers of all registered clients corresponding to the target file returned by the monitoring server; the target client is all registered clients except the client corresponding to the target file.
In the embodiment of the invention, the client saves the client ID returned by the monitoring server and the identification of the registered client corresponding to the target file, and determines the target client according to the registered client IP corresponding to the target file, wherein the target client does not comprise the newly registered client.
Optionally, in practical application, in order to ensure that only one client is allowed to modify a specific range of the target file at the same time, a corresponding write permission needs to be applied before modifying the target file. Referring to fig. 5, the substeps of step S10 may include:
step S103, when the client modifies the target file, a write permission request of the target file is sent to the monitoring server, so that the monitoring server returns authorization information according to the write permission request; the authorization information includes the identification of all registered clients corresponding to the target file.
When a client writes data to a certain page of a target file, firstly judging whether the client has the write permission of the page range lock of the target file, if the client has the write permission, directly writing the data to be written into a cache, and synchronizing the data to be written into the target client; if the write permission does not exist, a request for applying for the write permission of the target file is sent to the monitoring server, and the write permission request comprises a client ID, a file inode number and a page code.
The monitoring server inquires a lock authority table according to a file inode number and a page code in the received write authority request, and if the write authority of the range lock is not authorized to other clients, the holder of the range lock is directly updated to be a newly applied client; if the write permission of the range lock is authorized to other clients, sending a message for recalling the write permission of the lock to the client holding the write permission of the range lock, and updating the holder of the range lock to the newly applied client after successful recall of the write permission of the range lock. And simultaneously, acquiring the identifiers of all registered clients corresponding to the target file from the registry according to the file inode number, and sending the identifiers of all registered clients corresponding to the target file and the authorization result to the clients as authorization information.
Therefore, the client cannot actively release the write permission after modifying the data, the write permission is continuously held, and interaction between the client and the monitoring server is reduced in a mode of recalling the write permission by the monitoring server, so that the realization process of the lock is simplified, and the data modification efficiency is improved.
It should be noted that, the act of eliminating the identifier of the new application client from the identifiers of the registered clients may be performed by the monitoring server or may be performed by the client. The present application is not limited to this. After the client is granted write permission to the target file range lock, the page may be written.
Step S104, determining a target client according to the identifiers of all registered clients corresponding to the target file; the target client is all registered clients except the client corresponding to the target file.
In the embodiment of the invention, the target client is determined according to the identification of the registered clients except the client.
In order to ensure the integrity of the data read by the client, the local cache read-write mutual exclusion scene of the client is processed by using the local read-write lock of the client, namely the local cache of the client can be modified by holding the local write lock or the local cache data can be read by holding the local read lock, so that the data before partial update and the data after partial update are effectively prevented from being read.
Based on the same inventive concept, the embodiment of the invention also provides a data synchronization device. The basic principle and the technical effects are the same as those of the above embodiments, and for brevity, reference is made to the corresponding matters in the above embodiments where the description of the present embodiment is omitted.
Referring to fig. 6, fig. 6 is a block diagram illustrating a data synchronization apparatus 200 according to an embodiment of the invention. The data synchronization device 200 is applied to a client, and the data synchronization device 200 includes a decision module 201 and a synchronization module 202.
A decision module 201, configured to determine a target client that performs data synchronization with the client; the target client and the client are registered clients corresponding to the same target file.
And the synchronization module 202 is configured to perform dirty data synchronization with the target client, so that the client is consistent with the cached data of the target client.
In summary, the data synchronization device provided by the embodiment of the present invention includes a decision module and a synchronization module, where the decision module is configured to determine a target client that performs data synchronization with the client; the target client and the client are registered clients corresponding to the same target file. The synchronization module is used for carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client. The invention adopts a mode of directly synchronizing dirty data by a plurality of clients to ensure the consistency of client cache, reduces data interaction with a data server, effectively reduces the service burden of the data server and reduces the time of data synchronization, thereby improving IO read-write performance of a distributed file system.
Optionally, the synchronization module 202 is specifically configured to synchronize dirty data of the target client to the cache of the client when the client opens the target file; when the client modifies the target file, dirty data of the client is synchronized to the cache of the target client.
Optionally, the synchronization module 202 is specifically configured to send a dirty data query request of the target file to the target client when the client opens the target file; receiving dirty data of a target file returned by a target client based on a dirty data query request; and storing all dirty data of the received target file into a cache of the client.
Optionally, the synchronization module 202 is specifically configured to store, when the client modifies the target file, the data to be written as dirty data to the cache of the client; the dirty data is sent to the target client so that the target client saves the dirty data of the client to the cache.
Optionally, the decision module 201 is specifically configured to send a registration request to the monitoring server when the client opens the target file, so that the monitoring server obtains the identifiers of all registered clients corresponding to the target file according to the registration request; determining a target client according to the identifiers of all registered clients corresponding to the target file returned by the monitoring server; the target client is all registered clients except the client corresponding to the target file.
Optionally, the decision module 201 is specifically configured to send a write permission request of the target file to the monitoring server when the client modifies the target file, so that the monitoring server returns authorization information according to the write permission request; the authorization information comprises the identifications of all registered clients corresponding to the target file; determining a target client according to the identifiers of all registered clients corresponding to the target file; the target client is all registered clients except the client corresponding to the target file.
Referring to fig. 7, fig. 7 is a block diagram of a client 10 according to an embodiment of the invention. The client 10 may be any device having a data processing function, such as a personal computer (personal computer, PC), a palm top computer (Personal Digital Assistant, PDA), a notebook computer, a tablet computer, etc. The client 10 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
Wherein the memory 110 is used for storing programs or data. The Memory 110 may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
The processor 120 is used to read/write data or programs stored in the memory 110 and perform corresponding functions. For example, the data synchronization methods disclosed in the above embodiments may be implemented when a computer program stored in the memory 110 is executed by the processor 120.
The communication module 130 is used for establishing a communication connection between the client 10 and other communication terminals through a network, and for transceiving data through the network.
It should be understood that the structure shown in fig. 7 is merely a schematic diagram of the structure of the client 10, and that the client 10 may also include more or fewer components than shown in fig. 7, or have a different configuration than shown in fig. 7. The components shown in fig. 7 may be implemented in hardware, software, or a combination thereof.
Embodiments of the present invention also provide a computer readable storage medium having stored thereon a computer program which, when executed by the processor 120, implements the data synchronization method disclosed in the above embodiments.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present invention may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data synchronization, for application to a client, the method comprising:
determining a target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file;
and carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client.
2. The data synchronization method according to claim 1, wherein the performing dirty data synchronization with the target client to make the client consistent with the cached data of the target client includes:
when the client opens the target file, synchronizing dirty data of the target client to a cache of the client;
and synchronizing dirty data of the client to a cache of the target client when the client modifies the target file.
3. The method for synchronizing data according to claim 2, wherein synchronizing dirty data of the target client to the client's cache when the client opens the target file comprises:
when the client opens the target file, sending a dirty data query request of the target file to the target client;
receiving dirty data of the target file returned by the target client based on the dirty data query request;
and storing all the received dirty data of the target file into a cache of the client.
4. The data synchronization method according to claim 2, wherein synchronizing dirty data of the client to the cache of the target client when the client modifies the target file comprises:
when the client modifies the target file, the data to be written is stored as dirty data into a cache of the client;
and sending the dirty data to the target client so that the target client can store the dirty data of the client into a cache.
5. The method for data synchronization according to claim 1, wherein the determining a target client for data synchronization with the client includes:
when the client opens the target file, a registration request is sent to a monitoring server, so that the monitoring server obtains the identifiers of all registered clients corresponding to the target file according to the registration request;
determining a target client according to the identifiers of all registered clients corresponding to the target file returned by the monitoring server; and the target client is all registered clients except the client corresponding to the target file.
6. The method for data synchronization according to claim 1, wherein the determining a target client for data synchronization with the client includes:
when the client modifies the target file, sending a write permission request of the target file to a monitoring server so that the monitoring server returns authorization information according to the write permission request; the authorization information comprises the identifications of all registered clients corresponding to the target file;
determining a target client according to the identifiers of all registered clients corresponding to the target file; and the target client is all registered clients except the client corresponding to the target file.
7. A data synchronization device for application to a client, the device comprising:
the decision module is used for determining a target client for data synchronization with the client; the target client and the client are registered clients corresponding to the same target file;
and the synchronization module is used for carrying out dirty data synchronization with the target client so as to enable the client to be consistent with the cache data of the target client.
8. The data synchronization device of claim 7, wherein the synchronization module is configured to synchronize dirty data of the target client to a cache of the client when the client opens the target file; and synchronizing dirty data of the client to a cache of the target client when the client modifies the target file.
9. A client comprising a memory for storing a computer program and a processor for performing the data synchronization method according to any of claims 1-6 when the computer program is invoked.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements a data synchronization method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310834977.2A CN116561089B (en) | 2023-07-10 | 2023-07-10 | Data synchronization method, device, client and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310834977.2A CN116561089B (en) | 2023-07-10 | 2023-07-10 | Data synchronization method, device, client and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116561089A true CN116561089A (en) | 2023-08-08 |
| CN116561089B CN116561089B (en) | 2023-09-19 |
Family
ID=87496913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310834977.2A Active CN116561089B (en) | 2023-07-10 | 2023-07-10 | Data synchronization method, device, client and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116561089B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562477A (en) * | 2008-04-15 | 2009-10-21 | 北京易路联动技术有限公司 | Method and system for time management, client and server based on mobile internet |
| CN101771723A (en) * | 2010-01-05 | 2010-07-07 | 深圳中兴网信科技有限公司 | Data synchronization method |
| CN102014158A (en) * | 2010-11-29 | 2011-04-13 | 北京兴宇中科科技开发股份有限公司 | Cloud storage service client high-efficiency fine-granularity data caching system and method |
| CN102023816A (en) * | 2010-11-04 | 2011-04-20 | 天津曙光计算机产业有限公司 | Object storage policy and access method of object storage system |
| CN102541983A (en) * | 2011-10-25 | 2012-07-04 | 无锡城市云计算中心有限公司 | Method for synchronously caching by multiple clients in distributed file system |
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN102882927A (en) * | 2012-08-29 | 2013-01-16 | 华南理工大学 | Cloud storage data synchronizing framework and implementing method thereof |
| CN102984256A (en) * | 2012-11-28 | 2013-03-20 | 中国科学院计算技术研究所 | Processing method and system for metadata based on authorization manner |
| CN103167026A (en) * | 2013-02-06 | 2013-06-19 | 数码辰星科技发展（北京）有限公司 | Processing method, system and device for cloud storage environmental data |
| CN103200211A (en) * | 2012-01-04 | 2013-07-10 | 中国移动通信集团公司 | Method, system and equipment of data synchronism |
| CN103679048A (en) * | 2012-09-20 | 2014-03-26 | Sap股份公司 | Systems and methods for data privacy and destruction in multi-system landscapes |
| CN107168642A (en) * | 2017-03-30 | 2017-09-15 | 北京奇艺世纪科技有限公司 | A kind of date storage method and system |
| US20180341693A1 (en) * | 2016-01-18 | 2018-11-29 | Alibaba Group Holding Limited | Data synchronization method, apparatus, and system |
| CN111448559A (en) * | 2017-12-28 | 2020-07-24 | 卓普网盘股份有限公司 | Efficient management of client-side synchronization updates |
| CN111813788A (en) * | 2020-05-20 | 2020-10-23 | 北京金山云网络技术有限公司 | Information query method and device and information synchronization method and device |
-
2023
- 2023-07-10 CN CN202310834977.2A patent/CN116561089B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562477A (en) * | 2008-04-15 | 2009-10-21 | 北京易路联动技术有限公司 | Method and system for time management, client and server based on mobile internet |
| CN101771723A (en) * | 2010-01-05 | 2010-07-07 | 深圳中兴网信科技有限公司 | Data synchronization method |
| CN102023816A (en) * | 2010-11-04 | 2011-04-20 | 天津曙光计算机产业有限公司 | Object storage policy and access method of object storage system |
| CN102014158A (en) * | 2010-11-29 | 2011-04-13 | 北京兴宇中科科技开发股份有限公司 | Cloud storage service client high-efficiency fine-granularity data caching system and method |
| CN102541983A (en) * | 2011-10-25 | 2012-07-04 | 无锡城市云计算中心有限公司 | Method for synchronously caching by multiple clients in distributed file system |
| CN103200211A (en) * | 2012-01-04 | 2013-07-10 | 中国移动通信集团公司 | Method, system and equipment of data synchronism |
| CN102882927A (en) * | 2012-08-29 | 2013-01-16 | 华南理工大学 | Cloud storage data synchronizing framework and implementing method thereof |
| CN103679048A (en) * | 2012-09-20 | 2014-03-26 | Sap股份公司 | Systems and methods for data privacy and destruction in multi-system landscapes |
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN102984256A (en) * | 2012-11-28 | 2013-03-20 | 中国科学院计算技术研究所 | Processing method and system for metadata based on authorization manner |
| CN103167026A (en) * | 2013-02-06 | 2013-06-19 | 数码辰星科技发展（北京）有限公司 | Processing method, system and device for cloud storage environmental data |
| US20180341693A1 (en) * | 2016-01-18 | 2018-11-29 | Alibaba Group Holding Limited | Data synchronization method, apparatus, and system |
| CN107168642A (en) * | 2017-03-30 | 2017-09-15 | 北京奇艺世纪科技有限公司 | A kind of date storage method and system |
| CN111448559A (en) * | 2017-12-28 | 2020-07-24 | 卓普网盘股份有限公司 | Efficient management of client-side synchronization updates |
| CN111813788A (en) * | 2020-05-20 | 2020-10-23 | 北京金山云网络技术有限公司 | Information query method and device and information synchronization method and device |
Non-Patent Citations (1)
| Title |
|---|
| 黄云祥 等: "分布式文件监控系统的设计与实现", 《企业科技与发展》, no. 08, pages 20 - 22 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116561089B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7266645B2 (en) | Reducing communication for reads and updates in distributed object systems | |
| US6944732B2 (en) | Method and apparatus for supporting snapshots with direct I/O in a storage area network | |
| JP5081631B2 (en) | Method and apparatus for managing data deletion | |
| US5819272A (en) | Record tracking in database replication | |
| US8843454B2 (en) | Elimination of duplicate objects in storage clusters | |
| US5175851A (en) | System and method for controlling client machine access to a portion of a file with a variable length | |
| CN106682186B (en) | File access control list management method and related device and system | |
| JP2008527571A5 (en) | ||
| CN106326229B (en) | File storage method and device of embedded system | |
| US11599503B2 (en) | Path name cache for notifications of file changes | |
| CN109144413A (en) | A kind of metadata management method and device | |
| CN112632375B (en) | Session information processing method, server and storage medium | |
| CN115277145A (en) | Distributed storage access authorization management method, system, device and readable medium | |
| CN116561089B (en) | Data synchronization method, device, client and computer readable storage medium | |
| CN112817962B (en) | Data storage method and device based on object storage and computer equipment | |
| CN104021137A (en) | Method and system for opening and closing file locally through client side based on catalogue authorization | |
| US20040243827A1 (en) | Method and system for managing access control | |
| CN111796772B (en) | Cache management method, cache node and distributed storage system | |
| CN113342507A (en) | Distributed lock service implementation method and device and computer equipment | |
| CN110554914B (en) | Resource lock management method, device, server and storage medium | |
| JPH04313126A (en) | File input/output system for decentralized file system | |
| CN110795031A (en) | Data deduplication method, device and system based on full flash storage | |
| CN112286448B (en) | Object access method and device, electronic equipment and machine-readable storage medium | |
| CN114116651A (en) | System and method for supporting multi-protocol unified management user to access files | |
| CN114528260A (en) | File access request processing method, electronic equipment and computer program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |