CN115858482A - Data synchronization method, system and storage medium based on file system - Google Patents
Data synchronization method, system and storage medium based on file system Download PDFInfo
- Publication number
- CN115858482A CN115858482A CN202211721305.2A CN202211721305A CN115858482A CN 115858482 A CN115858482 A CN 115858482A CN 202211721305 A CN202211721305 A CN 202211721305A CN 115858482 A CN115858482 A CN 115858482A
- Authority
- CN
- China
- Prior art keywords
- file system
- data block
- file
- module
- synchronized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of information, in particular to a data synchronization method, a data synchronization system and a storage medium based on a file system, wherein the method comprises the following steps: receiving files and/or directories needing to be synchronized, and receiving target computer information; acquiring the type of a file system where files and/or directories needing to be synchronized are/is located; analyzing the information of the file system according to the type of the file system, obtaining a data block where a file and/or a directory needing to be synchronized are/is located, taking the data block as an effective data block, and recording the position of the effective data block; synchronizing the effective data blocks to a target computer according to the positions of the effective data blocks, and generating a restored file system by the target computer; and the target computer analyzes and restores the file system, deletes files and/or directories which do not need to be synchronized, and completes data synchronization. The beneficial technical effects of the invention comprise: meanwhile, the requirements of rapidly synchronizing massive small file data and flexibly selecting specific files/directories for synchronization are met, and the synchronization time of massive small files is greatly shortened.
Description
Technical Field
The invention relates to the technical field of information, in particular to a data synchronization method and system based on a file system and a storage medium.
Background
With the development of information technology, more and more data are generated by each application system, and the requirement of data synchronization is also highlighted.
For unstructured data, the conventional synchronization scheme mainly traverses and reads data of a file system of a source computer through a file operation interface of an operating system, and then synchronizes the data to a destination computer through an IP network, an FC, and the like. Or block copy is made directly to the disk across a file manipulation interface provided by the operating system. For a file system with a large number of small files, frequent system calls and disk addressing are required in a file operation interface mode, and the requirement for quickly synchronizing data cannot be met. The data synchronization is performed by using a disk block copying mode, and a specific file/directory cannot be selected for copying, so that the flexibility is too low. For this reason, research is required to improve the file system-based data synchronization technology.
The prior art discloses a method and a system for realizing remote data synchronization of a file system, wherein the method comprises the following steps: collecting relevant information of first file system data of a local server through a synchronization agent deployed in the local server, and transmitting the relevant information of the first file system data and a preset data comparison algorithm to a data protection server deployed in a remote server; the data protection server compares the related information of the first file system data with the related information of the second file system data of the remote server through a data comparison algorithm, and feeds back the comparison result to the synchronization agent when the comparison result shows that the related information of the first file system data is different from the related information of the second file system data; the synchronization agent performs data synchronization between the local server and the remote server according to the comparison result. By the scheme of the embodiment, remote protection of the file system data under a low-real-time and low-quality network is realized. However, the technical scheme of the method cannot solve the problem that the data synchronization efficiency based on the file system is low or the flexibility of the data synchronization based on the hard disk block is low at present.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: at present, the data synchronization efficiency based on a file system is low or the data synchronization flexibility based on a hard disk block is low. A data synchronization method, a data synchronization system and a storage medium based on a file system are provided, which can take account of low efficiency or flexibility of data synchronization.
In order to solve the technical problem, the invention adopts the following technical scheme: a data synchronization method based on a file system comprises the following steps:
receiving files and/or directories needing to be synchronized, and receiving target computer information;
acquiring the type of a file system where files and/or directories needing to be synchronized are/is located;
analyzing the information of the file system according to the type of the file system, obtaining a data block where a file and/or a directory needing to be synchronized is located, taking the data block as an effective data block, and recording the position of the effective data block;
synchronizing the valid data blocks to the target computer according to the valid data block positions, and generating a restored file system by the target computer;
and the target computer analyzes the restored file system, deletes files and/or directories which do not need to be synchronized, and completes data synchronization.
Preferably, the method for analyzing the information of the file system includes:
analyzing the metadata of the file system;
acquiring file information of the file system according to the metadata;
and obtaining the effective data block by the file information and the file and/or the directory which need to be synchronized.
Preferably, the method for parsing the metadata of the file system includes:
establishing a plurality of file system analysis modules, wherein the file system analysis modules respectively correspond to different file system types;
and according to the file system type, selecting a corresponding file system analysis module to analyze the file system to obtain metadata of the file system.
Preferably, the method of recording the location of the valid data block includes:
establishing a bitmap, wherein the bit quantity of the bitmap is consistent with the data block quantity of the file system;
if the data block is an effective data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not an effective data block, the bit value of the corresponding bitmap is 0.
Preferably, the method for generating the restored file system by the target computer comprises the following steps:
storing all the valid data blocks;
reading metadata contained in the effective data block;
and establishing indexes of files and/or directories contained in the effective database according to the metadata to complete the generation of a restored file system.
A data synchronization system based on a file system is used for executing the data synchronization method based on the file system, and comprises an input module, a detection module, a file system analysis module, an effective data block recording module, a data transmission module and a metadata clipping module, wherein the input module, the detection module, the file system analysis module, the effective data block recording module and the data transmission module are all deployed on a computer where a data source is located, the metadata clipping module is deployed on a target computer,
the input module receives files and/or directories to be synchronized, and receives target computer information,
the probing module obtains the type of file system where the files and/or directories to be synchronized are located,
the file system analysis module analyzes the information of the file system according to the type of the file system to obtain a data block where a file and/or a directory which needs to be synchronized is located as an effective data block, the effective data block recording module records the position of the effective data block, the data transmission module synchronizes the effective data block to the target computer according to the position of the effective data block, the metadata clipping module generates a restored file system, the metadata clipping module analyzes the restored file system, deletes files and/or directories which do not need to be synchronized, and data synchronization is completed.
Preferably, when the file system analysis module analyzes the information of the file system, the following steps are executed:
analyzing metadata of the file system;
acquiring file information of the file system according to the metadata;
and obtaining the effective data block by the file information and the file and/or the directory which need to be synchronized.
Preferably, when the valid data block recording module records the position of the valid data block, the following steps are executed:
establishing a bitmap, wherein the bit quantity of the bitmap is consistent with the data block quantity of the file system;
if the data block is an effective data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not an effective data block, the bit value of the corresponding bitmap is 0.
A computer system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the computer program when executed by the processor implementing a file system based data synchronization method as described above.
A computer-readable storage medium, storing a computer program which, when executed by a computer system, implements a file system based data synchronization method as described above.
The beneficial technical effects of the invention comprise: the method and the device realize identification and analysis of the file system, analyze the data of the selected file/directory and the disk block where the metadata is located, synchronize the corresponding disk block, finally meet the requirements of rapidly synchronizing massive small file data and flexibly selecting a specific file/directory for synchronization, and greatly shorten the synchronization time of massive small files.
Other features and advantages of the present invention will be disclosed in more detail in the following detailed description of the invention and the accompanying drawings.
Drawings
The invention is further described with reference to the accompanying drawings:
fig. 1 is a schematic flow chart of a data synchronization method according to an embodiment of the present invention.
Fig. 2 is a schematic flow chart of a method for analyzing file system information according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method for generating a restored file system by a target computer according to an embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention.
FIG. 5 is a block diagram of a computer system according to an embodiment of the present invention.
Wherein: 10. the system comprises an input module, 20, a detection module, 30, a file system analysis module, 40, an effective data block recording module, 50, a data transmission module, 60, a metadata clipping module, 70, a computer system, 71, a memory, 72, a computer program, 73 and a processor.
Detailed Description
The technical solutions of the embodiments of the present invention are explained and illustrated below with reference to the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, and not all embodiments. Based on the embodiments in the implementation, other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present invention.
In the following description, the appearances of the indicating orientation or positional relationship such as the terms "inner", "outer", "upper", "lower", "left", "right", etc. are only for convenience in describing the embodiments and for simplicity in description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and are not to be construed as limiting the present invention.
A data synchronization method based on a file system, referring to fig. 1, comprising the following steps:
step A01) receiving files and/or directories needing synchronization and receiving target computer information;
step A02) obtaining the file system type of the file and/or the directory needing to be synchronized;
step A03) according to the type of the file system, analyzing the information of the file system, obtaining a data block where a file and/or a directory needing to be synchronized is located, taking the data block as an effective data block, and recording the position of the effective data block;
step A04) synchronizing the effective data blocks to a target computer according to the positions of the effective data blocks, and generating a restored file system by the target computer;
step A05), the target computer analyzes and restores the file system, deletes files and/or directories which do not need to be synchronized, and completes data synchronization.
The file system is identified and analyzed, the data of the selected file/directory and the disk block where the metadata is located are analyzed, the corresponding disk blocks are synchronized, finally, the requirements of quickly synchronizing massive small file data and flexibly selecting a specific file/directory for synchronization are met, and the synchronization time of massive small files is greatly shortened.
Specifically, the method comprises the following steps: the user selects the files/directories to be synchronized, as well as the destination computer information.
The file system type detection module 20 selects a corresponding file system analysis module 30 according to the file system type of the file/directory selected by the user.
The file system parsing module 30 parses basic information (such as block size, etc.) of the file system, metadata location, data block location where the file/directory selected by the user is located and records location information thereof through the "valid data block recording and arranging module".
The effective data block recording and arranging module: according to the analysis of a file system analysis module, a bitmap is used for recording data blocks needing to be synchronized, 1 is used for indicating that the corresponding data blocks need to be synchronized, 0 is used for indicating that the corresponding data blocks do not need to be synchronized, the module is used for reading and sequencing the data blocks to be synchronized, one bit in the bitmap corresponds to one data block in a disk, the first bit in the bitmap corresponds to the first data block in the disk, the second bit corresponds to the second data block in the disk, and the like.
The data transmission module 50: and reading the data blocks needing synchronization to a destination computer according to the bitmap sequence generated by the effective data block recording and arranging module to generate a file system. It can be written to a physical hard disk, block device, or sparse file.
After the data transmission is completed, the metadata clipping module 60 analyzes the generated file system according to the file/directory selected by the user, and clips redundant metadata information, so that the file system of the destination computer only contains the file/directory selected by the user.
On the other hand, the embodiment provides a method for analyzing information of a file system, please refer to fig. 2, which includes:
step B01) analyzing metadata of the file system;
step B02) obtaining file information of the file system according to the metadata;
step B03) is to obtain the effective data block by the file information and the file and/or the directory which need to be synchronized.
The method for analyzing the metadata of the file system comprises the following steps: establishing a plurality of file system analysis modules 30, wherein the file system analysis modules 30 respectively correspond to different file system types; according to the type of the file system, the corresponding file system analysis module 30 is selected to analyze the file system to obtain the metadata of the file system.
The method for recording the position of the effective data block comprises the following steps: establishing a bitmap, wherein the number of bits of the bitmap is consistent with the number of data blocks of the file system; if the data block is a valid data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not a valid data block, the bit value of the corresponding bitmap is 0.
In another aspect, this embodiment provides a method for generating a restored file system by a target computer, referring to fig. 3, including:
step C01) storing all effective data blocks;
step C02) reading metadata contained in the valid data block;
and step C03) establishing indexes of the files and/or directories contained in the effective database according to the metadata to finish the generation of the restored file system.
On the other hand, the embodiment provides a data synchronization system based on a file system, which is used for executing the aforementioned data synchronization method based on a file system, and referring to fig. 4, the data synchronization system includes an input module 10, a detection module 20, a file system parsing module 30, an effective data block recording module 40, a data transmission module 50, and a metadata clipping module 60, the input module 10, the detection module 20, the file system parsing module 30, the effective data block recording module 40, and the data transmission module 50 are all deployed in a computer where a data source is located, the metadata clipping module 60 is deployed in a target computer,
the input module 10 receives files and/or directories that need to be synchronized, and receives target computer information,
the probing module 20 obtains the type of file system in which the files and/or directories that need to be synchronized are located,
the file system analysis module 30 analyzes the information of the file system according to the type of the file system to obtain the data block where the file and/or the directory which need to be synchronized are located as the valid data block, the valid data block recording module 40 records the position of the valid data block, the data transmission module 50 synchronizes the valid data block to the target computer according to the position of the valid data block, the metadata clipping module 60 generates a restored file system, the metadata clipping module 60 analyzes the restored file system, and deletes the file and/or the directory which do not need to be synchronized to complete data synchronization.
When the file system parsing module 30 parses the information of the file system, the following steps are performed:
analyzing metadata of a file system;
acquiring file information of the file system according to the metadata;
and obtaining the effective data block by the file information and the file and/or the directory needing to be synchronized.
When the valid data block recording module 40 records the position of the valid data block, the following steps are executed:
establishing a bitmap, wherein the number of bits of the bitmap is consistent with the number of data blocks of the file system;
if the data block is a valid data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not a valid data block, the bit value of the corresponding bitmap is 0.
In another aspect, the present invention provides a computer system, and referring to fig. 5, a computer system 70 includes a memory 71, a processor 73, and a computer program 72 stored in the memory 71 and executable on the processor 73, wherein the computer program 72 implements a file system based data synchronization method as described above when executed by the processor 73.
The computer system may be a general purpose computer system or a special purpose computer system. In particular implementations, the computer system may be a server cluster including a plurality of servers, such as may be a blockchain system including a plurality of nodes. Those skilled in the art will appreciate that fig. 5 is merely exemplary of a computer system and is not intended to limit the computer system and that many more or less components than those shown, or some components in combination, or different components such as input output devices, network access devices, etc., may be included.
The Processor 73 may be a Central Processing Unit (CPU), and the Processor 73 may also be other general-purpose Processor 73, a Digital Signal Processor 73 (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor 73 may be a microprocessor 73 or may be any conventional processor 73.
The storage 71 may in some embodiments be an internal storage unit of the computer system, such as a hard disk or a memory of the computer system. The memory 71 may also be an external storage device of the computer system in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the computer system. Further, the memory 71 may also include both internal storage units of the computer system and external storage devices. The memory 71 is used to store an operating system, an application program, a Boot Loader (Boot Loader), data, and other programs. The memory 71 may also be used to temporarily store data that has been output or is to be output.
In another aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program 72, and when the computer program 72 is executed by a processor 73, the computer program 72 implements a data synchronization method based on a file system as described above.
While the invention has been described with reference to specific embodiments thereof, it will be understood by those skilled in the art that the invention is not limited thereto, and may be embodied in many different forms without departing from the spirit and scope of the invention as set forth in the following claims. Any modification which does not depart from the functional and structural principles of the present invention is intended to be included within the scope of the claims.
Claims (10)
1. A data synchronization method based on file system is characterized in that,
the method comprises the following steps:
receiving files and/or directories needing to be synchronized, and receiving target computer information;
acquiring the type of a file system where files and/or directories needing to be synchronized are/is located;
analyzing the information of the file system according to the type of the file system, obtaining a data block where a file and/or a directory needing to be synchronized is located, taking the data block as an effective data block, and recording the position of the effective data block;
synchronizing the valid data blocks to the target computer according to the valid data block positions, and generating a restored file system by the target computer;
and the target computer analyzes the restored file system, deletes files and/or directories which do not need to be synchronized, and completes data synchronization.
2. A data synchronization method based on file system as claimed in claim 1,
the method for analyzing the information of the file system comprises the following steps:
analyzing the metadata of the file system;
acquiring file information of the file system according to the metadata;
and obtaining the effective data block by the file information and the file and/or the directory which need to be synchronized.
3. The data synchronization method based on file system as claimed in claim 2,
the method for analyzing the metadata of the file system comprises the following steps:
establishing a plurality of file system analysis modules, wherein the file system analysis modules respectively correspond to different file system types;
and according to the file system type, selecting a corresponding file system analysis module to analyze the file system to obtain the metadata of the file system.
4. A method for file system based data synchronization according to any of the claims 1 to 3,
the method for recording the position of the effective data block comprises the following steps:
establishing a bitmap, wherein the bit quantity of the bitmap is consistent with the data block quantity of the file system;
if the data block is an effective data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not an effective data block, the bit value of the corresponding bitmap is 0.
5. A method for file system based data synchronization according to any of the claims 1 to 3,
the method for generating the restoration file system by the target computer comprises the following steps:
storing all the valid data blocks;
reading metadata contained in the effective data block;
and establishing indexes of files and/or directories contained in the effective database according to the metadata to complete the generation of a restored file system.
6. A file system based data synchronization system for performing a file system based data synchronization method according to any of claims 1 to 5,
comprises an input module, a detection module, a file system analysis module, an effective data block recording module, a data transmission module and a metadata cutting module, wherein the input module receiving module, the detection module, the file system analysis module, the effective data block recording module and the data transmission module are all deployed in a computer where a data source is located, the metadata cutting module is deployed in a target computer,
the input module receives files and/or directories to be synchronized, and receives target computer information,
the probing module obtains the type of file system where the files and/or directories to be synchronized are located,
the file system analysis module analyzes the information of the file system according to the type of the file system to obtain a data block where a file and/or a directory which needs to be synchronized is located as an effective data block, the effective data block recording module records the position of the effective data block, the data transmission module synchronizes the effective data block to the target computer according to the position of the effective data block, the metadata clipping module generates a restored file system, the metadata clipping module analyzes the restored file system, deletes files and/or directories which do not need to be synchronized, and data synchronization is completed.
7. A file system based data synchronization system according to claim 6,
when the file system analysis module analyzes the information of the file system, the following steps are executed:
analyzing the metadata of the file system;
obtaining file information of the file system according to the metadata;
and obtaining the effective data block by the file information and the file and/or the directory which need to be synchronized.
8. A file system based data synchronization system according to claim 6 or 7,
when the effective data block recording module records the position of the effective data block, the following steps are executed:
establishing a bitmap, wherein the bit quantity of the bitmap is consistent with the data block quantity of the file system;
if the data block is an effective data block, the bit value of the corresponding bitmap is 1, otherwise, if the data block is not an effective data block, the bit value of the corresponding bitmap is 0.
9. A computer system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the computer program when executed by the processor implementing a method of file system based data synchronization as claimed in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a computer system, implements a file system based data synchronization method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721305.2A CN115858482A (en) | 2022-12-30 | 2022-12-30 | Data synchronization method, system and storage medium based on file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721305.2A CN115858482A (en) | 2022-12-30 | 2022-12-30 | Data synchronization method, system and storage medium based on file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115858482A true CN115858482A (en) | 2023-03-28 |
Family
ID=85656317
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211721305.2A Pending CN115858482A (en) | 2022-12-30 | 2022-12-30 | Data synchronization method, system and storage medium based on file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115858482A (en) |
-
2022
- 2022-12-30 CN CN202211721305.2A patent/CN115858482A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| KR20150125012A (en) | Managing operations on stored data units | |
| WO2014067240A1 (en) | Method and apparatus for recovering sqlite file deleted from mobile terminal | |
| US20160034201A1 (en) | Managing de-duplication using estimated benefits | |
| CN109885577B (en) | Data processing method, device, terminal and storage medium | |
| EP3438845A1 (en) | Data updating method and device for a distributed database system | |
| KR20150125010A (en) | Managing operations on stored data units | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| CN115858488A (en) | Parallel migration method and device based on data governance and readable medium | |
| KR102275240B1 (en) | Managing operations on stored data units | |
| Billard et al. | Making sense of unstructured flash-memory dumps | |
| CN110417892B (en) | Message analysis-based data replication link optimization method and device | |
| CN113419897A (en) | File processing method and device, electronic equipment and storage medium thereof | |
| CN109902070B (en) | WiFi log data-oriented analysis storage search method | |
| CN111309677A (en) | File management method and device of distributed file system | |
| CN115858482A (en) | Data synchronization method, system and storage medium based on file system | |
| CN104063377A (en) | Information processing method and electronic equipment using same | |
| CN115328696A (en) | Data backup method in database | |
| CN114896276A (en) | Data storage method and device, electronic equipment and distributed storage system | |
| US20130218851A1 (en) | Storage system, data management device, method and program | |
| CN114443595A (en) | Method and device for processing file | |
| CN114297196A (en) | Metadata storage method and device, electronic equipment and storage medium | |
| WO2020211371A1 (en) | Image restoration method and apparatus, device and storage medium | |
| CN112948494A (en) | Data synchronization method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |