CN110297598B - Data synchronization method and storage system - Google Patents
Data synchronization method and storage system Download PDFInfo
- Publication number
- CN110297598B CN110297598B CN201810246361.2A CN201810246361A CN110297598B CN 110297598 B CN110297598 B CN 110297598B CN 201810246361 A CN201810246361 A CN 201810246361A CN 110297598 B CN110297598 B CN 110297598B
- Authority
- CN
- China
- Prior art keywords
- data
- storage system
- metadata
- entity data
- entity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data synchronization method and a storage system, which are used for solving the technical problem of low efficiency of the storage system in the prior art when the data synchronization is realized. The first storage system firstly acquires a synchronization task which is sent by the second storage system and comprises metadata, and the metadata indicates data type information and data access address information of entity data to be synchronized, so that the first storage system can acquire the entity data from the second storage system according to the metadata, and even if a user accesses the entity data through the first storage system in the synchronization process of the entity data, the first storage system can call the entity data from the second storage system according to the metadata and feed the entity data back to the user, and the efficiency of the storage system in realizing data synchronization is improved.
Description
Technical Field
The invention relates to the field of cloud storage, in particular to a data synchronization method and a storage system.
Background
In the prior art, in terms of achieving data consistency across regions and multiple resource pools (storage systems), a final consistency scheme is generally adopted, when resource pools are built, multiple resource pools needing to guarantee consistency of the multiple resource pools are configured, and after the multiple resource pools are used online, relevant rules are not allowed to be adjusted.
When the data of the source resource pool in the resource pools is successfully written, pushing the data change message to a target resource pool in the resource pools; and after receiving the data change message, the target resource pool pulls the data written into the source resource pool from the source resource pool, so that the data consistency of the source resource pool and the target resource pool is realized. However, in general, it takes a long time to achieve the consistency of the data of the source resource pool and the target resource pool, and therefore, the user cannot access the data in the target resource pool during this period, that is, when the data written into the source resource pool is not synchronized to the target resource pool, the user cannot sense or acquire the data from the target resource pool.
In summary, the storage system in the prior art has low efficiency in implementing data synchronization.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method and a storage system, which are used for solving the technical problem of low efficiency of the storage system in the prior art when data synchronization is realized.
In a first aspect, an embodiment of the present invention provides a data synchronization method, applied to a first storage system, where the first storage system is connected to a second storage system, and the method includes:
acquiring a synchronization task for synchronizing entity data sent by a second storage system, wherein the synchronization task comprises data type information used for indicating the entity data and metadata of data access address information of the entity data in the second storage system;
generating a data pulling task based on the synchronization task, wherein the data pulling task is used for instructing the first storage system to call entity data corresponding to the metadata from the second storage system;
and acquiring the entity data based on the data pulling task.
In the embodiment of the invention, the first storage system firstly acquires the synchronization task which is sent by the second storage system and comprises the metadata, and the metadata indicates the data type information and the data access address information of the entity data to be synchronized, so that the first storage system can acquire the entity data from the second storage system according to the metadata, and even if a user accesses the entity data through the first storage system in the synchronization process of the entity data, the first storage system can also call the entity data from the second storage system according to the metadata and feed the entity data back to the user, thereby improving the efficiency of the storage system in realizing data synchronization.
In a possible implementation manner, while or after the entity data is acquired based on the data pull task, the first storage system may acquire a data access request of a user, and because the data access request is used to request a write operation and/or a read operation on the entity data corresponding to the metadata, if the entity data in the first storage system is not completely acquired, the data access request may be sent to the second storage system to receive the entity data corresponding to the metadata, which is fed back by the second storage system based on the data access request, so that a defect that the user cannot access the data in the first storage system during the period due to a long data synchronization time between the first storage system and the second storage system is effectively avoided.
In a possible implementation manner, after acquiring a data access request of a user, the first storage system may determine whether the entity data is damaged; and if the entity data are damaged, the data access request is sent to the second storage system, and the second storage system feeds back the entity data to the first storage system according to the metadata in the data access request, so that the defect that the data in the first storage system cannot be accessed is avoided.
In a second aspect, an embodiment of the present invention provides a data synchronization method, which is applied to a second storage system, where the second storage system is connected to a first storage system, and the method includes:
acquiring user data;
determining metadata and entity data from the user data based on a preset synchronization policy, wherein the metadata is used for indicating data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy comprises at least one of a storage space level, a storage system level and a user level;
generating a synchronization task including the metadata for synchronizing the entity data based on the metadata and the entity data;
sending the synchronization task to the first storage system;
receiving a data pulling task generated by the first storage system based on the synchronization task, wherein the data pulling task is used for calling entity data corresponding to the metadata from the second storage system;
and sending the entity data to the first storage system based on the data pulling task.
In the embodiment of the invention, the second storage system respectively carries out data synchronization of the metadata layer and the entity data layer, and the occupied storage space ratio of the metadata is small, so that the pressure on bandwidth can be reduced, the synchronization speed is high, the probability of data inconsistency when the first storage system and the second storage system carry out reading and/or accessing operations can be greatly reduced, and the condition that a user cannot access data is avoided because the metadata indicates the data access address of the entity data.
In one possible implementation manner, after sending the synchronization task to the first storage system, the method further includes:
acquiring a data access request sent by the first storage system, wherein the data access request is used for requesting to perform write operation and/or read operation on entity data corresponding to the metadata;
determining data access address information indicated by the metadata based on the data access request, and determining the entity data from the data access address;
and feeding back the entity data to the first storage system.
In a third aspect, an embodiment of the present invention provides a first storage system, where the first storage system is connected to a second storage system, and the first storage system includes:
a first obtaining module, configured to obtain a synchronization task for synchronizing an entity data sent by a second storage system, where the synchronization task includes data type information used to indicate the entity data and metadata of data access address information of the entity data in the second storage system;
a generating module, configured to generate a data pulling task based on the synchronization task, where the data pulling task is used to instruct the first storage system to retrieve entity data corresponding to the metadata from the second storage system;
and the second acquisition module is used for acquiring the entity data based on the data pulling task.
In one possible implementation, the first storage system further includes:
a third obtaining module, configured to obtain, while or after the second obtaining module obtains the entity data based on the data pull task, a data access request of a user, where the data access request is used to request to perform write operation and/or read operation on the entity data corresponding to the metadata;
if the entity data is not acquired, the data access request is sent to the second storage system;
receiving entity data corresponding to the metadata, fed back by the second storage system based on the data access request.
In one possible implementation, the first storage system further includes:
the judging module is used for judging whether the entity data in the first storage system is damaged or not after the third obtaining module obtains the data access request of the user;
and if the entity data is damaged, sending the data access request to the second storage system.
In a fourth aspect, an embodiment of the present invention provides a second storage system, where the second storage system is connected to a first storage system, and the second storage system includes:
the acquisition module is used for acquiring user data;
a determining module, configured to determine metadata and entity data from the user data based on a preset synchronization policy, where the metadata is used to indicate data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy includes at least one of a storage space level, a storage system level, and a user level;
a generating module for generating a synchronization task including the metadata for synchronizing the entity data based on the metadata and the entity data;
the first sending module is used for sending the synchronization task to the first storage system;
a receiving module, configured to receive a data pulling task generated by the first storage system based on the synchronization task, where the data pulling task is used to retrieve entity data corresponding to the metadata from the second storage system;
and the second sending module is used for sending the entity data to the first storage system based on the data pulling task.
In one possible implementation, the second storage system further includes:
a feedback module, configured to obtain a data access request sent by the first storage system after the first sending module sends the synchronization task to the first storage system, where the data access request is used to request to perform a write operation and/or a read operation on entity data corresponding to the metadata;
determining data access address information indicated by the metadata based on the data access request, and determining the entity data from the data access address;
and feeding back the entity data to the first storage system.
In a fifth aspect, an embodiment of the present invention provides a computer apparatus, including:
at least one processor, and
a memory communicatively coupled to the at least one processor, a communication interface;
wherein the memory stores instructions executable by the at least one processor, and the at least one processor performs the method of the first aspect or the second aspect using the communication interface by executing the instructions stored by the memory.
In a sixth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions that, when executed on a computer, cause the computer to perform the method according to the first or second aspect.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a diagram illustrating an implementation architecture of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a data synchronization method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a first storage system according to an embodiment of the invention;
FIG. 4 is a diagram illustrating a second storage system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
The data synchronization method provided by the embodiment of the invention can realize data consistency among a plurality of storage systems in the same region or across regions, wherein the storage systems can be cloud storage systems and the like. When a certain storage system of the multiple storage systems has data written therein, the storage system may serve as a source resource pool, and the other storage systems of the multiple storage systems may serve as target resource pools, that is, the source resource pool may synchronize the written data to the target resource pools, so as to ensure data consistency among the multiple storage systems. Of course, any of the plurality of storage systems may serve as a source resource pool, or may serve as a target resource pool.
For convenience of description, the target resource pool is referred to as a first storage system, the source resource pool is a second storage system, and the number of the first storage systems may be one or more. In the following, a data synchronization method according to an embodiment of the present invention is described by taking an example of implementing data consistency between two storage systems.
Fig. 1 is a diagram illustrating an implementation architecture of a data synchronization method according to an embodiment of the present invention. In fig. 1, the first storage system or the second storage system may include a gateway module, a plurality of storage devices, a database, a Push (Push) module, and a Pull (Pull) module, where the aforementioned modules may be connected to each other, and the second storage system may serve as a source resource pool of the first storage system.
The Pull module and the Push module may exist as independent modules (processes) in the first storage system or the second storage system, or may also exist as sub-processes or threads of the gateway module in the first storage system or the second storage system, which is not limited in the embodiment of the present invention.
In practical application, before data is written into the second storage system for the first time, a user or a system administrator may configure resource pool information, if data synchronization between specified storage systems needs to be achieved, and a synchronization rule may also be added, if data synchronization of specified types needs to be achieved, and the like, the user or the system administrator may store the configuration, synchronization rule, and other information of the resource pool in a plurality of storage systems, and of course, the information may be flexibly adjusted in the data synchronization process or after data synchronization, thereby avoiding the defects that in the prior art, when a configuration file is damaged, the recovery difficulty is large, and when a new storage system is added subsequently, the change difficulty is large.
It is to be understood that the terms "first," "second," and the like in the description of the embodiments of the invention are used for distinguishing between descriptions and not necessarily for describing a sequential or chronological order. "plurality" in the description of the embodiments of the present invention means two or more.
The term "and/or" in the embodiment of the present invention is only an association relationship describing an association object, and indicates that three relationships may exist, for example, a and/or B, and may indicate: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 2 is a schematic flow chart of a data synchronization method when data consistency between a first storage system and a second storage system is achieved according to an embodiment of the present invention. It should be noted that, steps S201 and S202 in the embodiment of the present invention are only for convenience of description, and are not used to limit the order of steps of the data synchronization method.
Referring to fig. 1 and fig. 2, in the embodiment of the present invention, at the side of the second storage system, an implementation flow of the data synchronization method may be described as follows:
s201: acquiring user data;
s202: determining metadata and entity data from user data based on a preset synchronization strategy;
s203: based on the metadata and the entity data, a synchronization task is generated that includes the metadata for synchronizing the entity data.
In S201, a user may write data into the second storage system through the gateway module in the second storage system, where the data written by the user, that is, the user data, may be uploaded data such as videos, pictures, and web browsing records; then, the gateway module in the second storage system may write the user data into the underlying storage device for saving, and record the user data in the database.
Then, S202 is entered, that is, metadata and entity data may be determined from the user data according to a preset synchronization policy, where the metadata may be used to indicate data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy includes at least one of a storage space level, a storage system level, and a user level.
In practical applications, the metadata may be data describing entity data, and may be used to describe data attributes, such as storage location, history data, resource lookup, file record, and the like, such as record, name, size, type, and the like related to the entity data, and the entity data is basic data of the user data.
For example, if the user data is a video segment uploaded by the user, the metadata may include a storage location, a name, a total duration of the video, a size of the video data, and the like of the video segment, and the entity data may be basic data constituting the video segment itself; alternatively, if the user data is a web browsing record of the user, the metadata may include a file record, a resource lookup, and the like, and the entity data may be some basic data of the web browsing record itself, and the like.
The preset synchronization policy may be custom set by a user or a system administrator. The user can perform storage space level cross-resource pool data synchronization setting, and can customize a data consistency rule, for example, the user can customize and set which storage device the second storage system stores to synchronize the data in the first storage system.
For example, the second storage system includes a storage device a and a storage device B, and in this case, the user may set to synchronize data stored in the storage device a to the first storage device.
A system administrator of the storage system may perform storage system-level cross-resource pool data synchronization setting, for example, may set a synchronization level of each storage system in the multiple storage systems, for example, 3 storage systems are respectively the storage system 1, the storage system 2, and the storage system 3, and then the storage system-level setting may be the storage system 1> the storage system 2> the storage system 3, and thus, assuming that the storage system 2 is a source resource pool, the storage system 2 may synchronize data to the storage system 1 first, and then to the storage system 3.
The system administrator may also perform user-level cross-resource pool data synchronization setting, for example, the system administrator may specify which user data of users may perform cross-resource pool data synchronization, and which user data of users may not perform cross-resource pool data synchronization.
After the second storage system determines the metadata and the entity data according to the preset synchronization policy, S203 may be entered, that is, a Push module in the second storage system may monitor a data record to be synchronized in a database, and then generate a synchronization task for synchronizing the entity data, which may include the metadata, according to the metadata and the entity data.
Furthermore, the Push module in the second storage system can send the synchronization task to the first storage system to trigger the first storage system to perform data synchronization.
In the embodiment of the present invention, on the first storage system side, an implementation flow of the data synchronization method may specifically include the following steps:
s204: acquiring a synchronization task for synchronizing entity data transmitted by a second storage system;
s205: generating a data pulling task based on the synchronization task;
s207: and acquiring entity data based on the data pulling task.
In S204, the gateway module of the first storage system may obtain a synchronization task for synchronizing an entity data sent by the Push module of the second storage system, and since the synchronization task carries metadata, the gateway module in the first storage system may write the metadata into the database for storage, and may update the state of the metadata to be in synchronization.
Since the metadata is equivalent to a placeholder in the first storage system, no additional storage space is occupied except for the database records, and the synchronization speed of the metadata is high. And because the metadata can indicate the data access address information of the entity data in the second storage system, even if a user writes a large file with a large data volume in the second storage system and even if the entity data needs a long time after the metadata is synchronized by the first storage system and the second storage system, the user can access the entity data stored in the second storage system through the first storage system during the period, thereby effectively avoiding the defect that the user cannot read and/or access the corresponding data on the first storage system within a period of time after the data writing of the second storage system is successful due to long data synchronization time of the first storage system and the second storage system.
After the first storage system obtains the synchronization task sent by the second storage system, S205 may be entered, and the gateway module of the second storage system may generate a data pulling task according to the synchronization task and store the data pulling task in the database, where the data pulling task may be used to instruct the first storage system to retrieve entity data corresponding to the metadata from the second storage system.
After monitoring the data pulling task in the database, the Pull module in the first storage system may send the data pulling task to the second storage system, that is, S206 is entered, and the second storage system receives the data pulling task sent by the first storage system and then sends the entity data to the first storage system according to the data pulling task.
And then, S207 is entered, that is, the first storage system may receive the entity data sent by the first storage system according to the data pulling task.
After a Pull module in the first storage system pulls entity data from the second storage system, the entity data can be written into a gateway module of the first storage system, metadata placeholders written before are automatically covered, and after data writing is completed, the Pull module updates a pulling task list and updates the data state to be an available state.
In the existing storage system, when data synchronization is performed, metadata and entity data are not distinguished, generally, the metadata and the entity data are considered as a whole, and a unified updating and storing method is adopted, so that accessible data cannot be provided for a user or an address accessible by the user is prompted when data written in the second storage system is successful or data in the synchronized first storage system is damaged or lost, and further, the user cannot read and/or access corresponding data on the first storage system.
Based on the above problems in the prior art, in the embodiment of the present invention, a hierarchical synchronization method is used to ensure data consistency of the first storage system and the second storage system, that is, data synchronization of a metadata layer and an entity data layer is performed separately, because the storage space occupied by metadata is small, pressure on bandwidth can be reduced, the synchronization speed is high, the probability of data inconsistency when the first storage system and the second storage system perform reading and/or accessing operations can be greatly reduced, and because metadata indicates a data access address of entity data, a situation that a user cannot access data is avoided.
In a possible implementation manner, while or after acquiring the entity data based on the data pulling task, the first storage system may acquire a data access request of a user, where the data access request is used to request to perform a write operation and/or a read operation on the entity data corresponding to the metadata, and then the first storage system may determine whether the entity data is completed synchronously, that is, whether the entity data is completed.
If the entity data in the first storage system are not completely acquired, the first storage system can send the data access request to the second storage system, namely, the first storage system automatically redirects the access request to the second storage system, and receives the entity data corresponding to the metadata and fed back by the second storage system based on the data access request, so as to feed back the entity data to the user.
Or, in a possible implementation manner, after the entity data is acquired based on the data pulling task, the first storage system may further determine whether the entity data in the first storage system is damaged, if the entity data is damaged, that is, when the entity data acquired in the first storage system is not available, the first storage system may send the data access request to the second storage system, and after the second storage system acquires the data access request, the second storage system may determine data access address information indicated by the metadata according to the data access request, determine the entity data from the data access address, and then feed back the entity data to the first storage system, so that the first storage system may receive the entity data corresponding to the metadata and fed back to the user.
That is to say, since the metadata record of the first storage system records the data access address of the data synchronization status entity data in the second storage system, if the data copy of the first storage system is unavailable, the access request can be automatically redirected to the second storage system, thereby implementing the access of the data.
Or after the data synchronization, if the entity data in the second storage system is unavailable, if a user requests to access the data in the second storage system, the access request may also be directed to the first storage system, thereby implementing the data access.
In summary, one or more technical solutions of the embodiments of the present invention have at least the following technical effects or advantages:
in the embodiment of the invention, the first storage system firstly acquires the synchronization task which is sent by the second storage system and comprises the metadata, and the metadata indicates the data type information and the data access address information of the entity data to be synchronized, so that the first storage system can acquire the entity data from the second storage system according to the metadata, and even if a user accesses the entity data through the first storage system in the synchronization process of the entity data, the first storage system can also call the entity data from the second storage system according to the metadata and feed the entity data back to the user, thereby improving the efficiency of the storage system in realizing data synchronization.
Based on the same inventive concept, please refer to fig. 3, an embodiment of the present invention provides a first storage system, where the first storage system is connected to a second storage system, and the first storage system includes:
a first obtaining module 31, configured to obtain a synchronization task for synchronizing an entity data sent by a second storage system, where the synchronization task includes data type information used to indicate the entity data and metadata of data access address information of the entity data in the second storage system;
a generating module 32, configured to generate a data pulling task based on the synchronization task, where the data pulling task is used to instruct the first storage system to retrieve entity data corresponding to the metadata from the second storage system;
a second obtaining module 33, configured to obtain the entity data based on the data pulling task.
In one possible implementation, the first storage system further includes:
a third obtaining module, configured to, while or after the second obtaining module 33 obtains the entity data based on the data pull task, obtain a data access request of a user, where the data access request is used to request to perform write operation and/or read operation on the entity data corresponding to the metadata;
if the entity data is not acquired, the data access request is sent to the second storage system;
receiving entity data corresponding to the metadata, fed back by the second storage system based on the data access request.
In one possible implementation, the first storage system further includes:
the judging module is used for judging whether the entity data in the first storage system is damaged or not after the third acquiring module acquires the data access request of the user;
and if the entity data is damaged, sending the data access request to the second storage system.
Based on the same inventive concept, please refer to fig. 4 again, an embodiment of the present invention provides a second storage system, where the second storage system is connected to a first storage system, and the second storage system includes:
an obtaining module 41, configured to obtain user data;
a determining module 42, configured to determine metadata and entity data from the user data based on a preset synchronization policy, where the metadata is used to indicate data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy includes at least one of a storage space level, a storage system level, and a user level;
a generating module 43, configured to generate a synchronization task including the metadata and used for synchronizing the entity data based on the metadata and the entity data;
a first sending module 44, configured to send the synchronization task to the first storage system;
a receiving module 45, configured to receive a data pulling task generated by the first storage system based on the synchronization task, where the data pulling task is used to retrieve entity data corresponding to the metadata from the second storage system;
a second sending module 46, configured to send the entity data to the first storage system based on the data pulling task.
In one possible implementation, the second storage system further includes:
a feedback module, configured to obtain a data access request sent by the first storage system after the first sending module 44 sends the synchronization task to the first storage system, where the data access request is used to request to perform a write operation and/or a read operation on entity data corresponding to the metadata;
determining data access address information indicated by the metadata based on the data access request, and determining the entity data from the data access address;
and feeding back the entity data to the first storage system.
Referring to fig. 5, based on the same inventive concept, an embodiment of the present invention provides a computer apparatus, which includes at least one processor 51, and a memory 52 and a communication interface 53 communicatively connected to the at least one processor 51, where fig. 5 illustrates one processor 51 as an example.
Wherein the memory 52 stores instructions executable by the at least one processor 51, and the at least one processor 51 executes the method described above by executing the instructions stored in the memory 52 using the communication interface 53.
Based on the same inventive concept, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described above.
In particular implementations, the computer-readable storage medium includes: a Universal Serial Bus flash drive (USB), a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other storage media capable of storing program codes.
The above-described embodiments of the apparatus are merely illustrative, wherein units/modules illustrated as separate components may or may not be physically separate, and components shown as units/modules may or may not be physical units/modules, may be located in one place, or may be distributed over a plurality of network units/modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A data synchronization method is applied to a first storage system, and is characterized in that the first storage system is connected with a second storage system, and the method comprises the following steps:
acquiring a synchronization task for synchronizing entity data sent by a second storage system, wherein the synchronization task comprises data type information used for indicating the entity data and metadata of data access address information of the entity data in the second storage system;
generating a data pulling task based on the synchronization task, wherein the data pulling task is used for instructing the first storage system to call entity data corresponding to the metadata from the second storage system;
acquiring the entity data based on the data pulling task;
while or after the entity data is acquired based on the data pulling task, the method further includes:
acquiring a data access request of a user, wherein the data access request is used for requesting to perform write operation and/or read operation on entity data corresponding to the metadata;
if the entity data is not acquired, the data access request is sent to the second storage system;
receiving entity data corresponding to the metadata, fed back by the second storage system based on the data access request.
2. The method of claim 1, wherein after obtaining the user's data access request, further comprising:
judging whether the entity data in the first storage system is damaged or not;
and if the entity data is damaged, sending the data access request to the second storage system.
3. A data synchronization method is applied to a second storage system, and the second storage system is connected with a first storage system, and the method comprises the following steps:
acquiring user data;
determining metadata and entity data from the user data based on a preset synchronization policy, wherein the metadata is used for indicating data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy comprises at least one of a storage space level, a storage system level and a user level;
generating a synchronization task including the metadata for synchronizing the entity data based on the metadata and the entity data;
sending the synchronization task to the first storage system;
receiving a data pulling task generated by the first storage system based on the synchronization task, wherein the data pulling task is used for calling entity data corresponding to the metadata from the second storage system;
sending the entity data to the first storage system based on the data pulling task;
after sending the synchronization task to the first storage system, further comprising:
acquiring a data access request sent by the first storage system, wherein the data access request is used for requesting to perform write operation and/or read operation on entity data corresponding to the metadata;
determining data access address information indicated by the metadata based on the data access request, and determining the entity data from the data access address;
and feeding back the entity data to the first storage system.
4. A first storage system, wherein the first storage system is connected to a second storage system, the first storage system comprising:
a first obtaining module, configured to obtain a synchronization task for synchronizing an entity data sent by a second storage system, where the synchronization task includes data type information used to indicate the entity data and metadata of data access address information of the entity data in the second storage system;
a generating module, configured to generate a data pulling task based on the synchronization task, where the data pulling task is used to instruct the first storage system to retrieve entity data corresponding to the metadata from the second storage system;
a second obtaining module, configured to obtain the entity data based on the data pulling task;
the first storage system further comprises:
a third obtaining module, configured to obtain a data access request of a user while or after the second obtaining module obtains the entity data based on the data pull task, where the data access request is used to request to perform write operation and/or read operation on the entity data corresponding to the metadata;
if the entity data is not acquired, the data access request is sent to the second storage system;
receiving entity data corresponding to the metadata, fed back by the second storage system based on the data access request.
5. The first storage system of claim 4, wherein the first storage system further comprises:
the judging module is used for judging whether the entity data in the first storage system is damaged or not after the third obtaining module obtains the data access request of the user;
and if the entity data is damaged, sending the data access request to the second storage system.
6. A second storage system, wherein the second storage system is connected to the first storage system, the second storage system comprising:
the acquisition module is used for acquiring user data;
a determining module, configured to determine metadata and entity data from the user data based on a preset synchronization policy, where the metadata is used to indicate data type information of the entity data and data access address information of the entity data in the second storage system, and the preset synchronization policy includes at least one of a storage space level, a storage system level, and a user level;
a generating module for generating a synchronization task including the metadata for synchronizing the entity data based on the metadata and the entity data;
the first sending module is used for sending the synchronization task to the first storage system;
a receiving module, configured to receive a data pulling task generated by the first storage system based on the synchronization task, where the data pulling task is used to retrieve entity data corresponding to the metadata from the second storage system;
a second sending module, configured to send the entity data to the first storage system based on the data pull task;
the second storage system further includes:
a feedback module, configured to obtain a data access request sent by the first storage system after the first sending module sends the synchronization task to the first storage system, where the data access request is used to request to perform write operation and/or read operation on entity data corresponding to the metadata;
determining data access address information indicated by the metadata based on the data access request, and determining the entity data from the data access address;
and feeding back the entity data to the first storage system.
7. A computer device, the computer device comprising:
at least one processor, and
a memory communicatively coupled to the at least one processor, a communication interface;
wherein the memory stores instructions executable by the at least one processor to perform the method of any one of claims 1-3 using the communication interface by executing the instructions stored by the memory.
8. A computer-readable storage medium having stored thereon computer instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810246361.2A CN110297598B (en) | 2018-03-23 | 2018-03-23 | Data synchronization method and storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810246361.2A CN110297598B (en) | 2018-03-23 | 2018-03-23 | Data synchronization method and storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110297598A CN110297598A (en) | 2019-10-01 |
| CN110297598B true CN110297598B (en) | 2022-12-13 |
Family
ID=68025953
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810246361.2A Active CN110297598B (en) | 2018-03-23 | 2018-03-23 | Data synchronization method and storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110297598B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103873451A (en) * | 2012-12-17 | 2014-06-18 | 中国移动通信集团公司 | Data synchronization method, equipment and system |
| CN103944709A (en) * | 2014-03-26 | 2014-07-23 | 小米科技有限责任公司 | Data synchronization method and device |
-
2018
- 2018-03-23 CN CN201810246361.2A patent/CN110297598B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103873451A (en) * | 2012-12-17 | 2014-06-18 | 中国移动通信集团公司 | Data synchronization method, equipment and system |
| CN103944709A (en) * | 2014-03-26 | 2014-07-23 | 小米科技有限责任公司 | Data synchronization method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110297598A (en) | 2019-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11221995B2 (en) | Data replication from a cloud-based storage resource | |
| US11188560B2 (en) | Synchronizing object in local object storage node | |
| US20150227605A1 (en) | Information processing terminal, synchronization control method, and computer-readable recording medium | |
| CN106469087B (en) | Metadata output method, client and metadata server | |
| US11093335B2 (en) | Data replication method, apparatus, and system | |
| WO2020098654A1 (en) | Data storage method and device based on cloud storage, and storage medium | |
| CN108491163B (en) | Big data processing method and device and storage medium | |
| US20120324436A1 (en) | Method of updating versioned software using a shared cache | |
| CN110865985B (en) | Data synchronization method, device, electronic equipment and storage medium | |
| US10795747B2 (en) | File synchronizing service status monitoring and error handling | |
| CN107040576A (en) | Information-pushing method and device, communication system | |
| CN105335450B (en) | Data storage processing method and device | |
| CN106339176B (en) | Intermediate file processing method, client, server and system | |
| CN111383038A (en) | Advertisement display method and device of mobile terminal, mobile terminal and storage medium | |
| CN108153794B (en) | Page cache data refreshing method, device and system | |
| CN113297316A (en) | Method, device and system for realizing data synchronization | |
| CN110297598B (en) | Data synchronization method and storage system | |
| CN111459619A (en) | Method and device for realizing service based on cloud platform | |
| CN116483274A (en) | Online migration method, device, equipment and medium for distributed block storage volume | |
| CN107145302B (en) | Method and equipment for executing file writing in distributed storage system | |
| CN111506254A (en) | Distributed storage system and management method and device thereof | |
| WO2018028321A1 (en) | Method and apparatus for managing virtual external storage device, and terminal | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN111147226B (en) | Data storage method, device and storage medium | |
| EP2960778A1 (en) | A storage abstraction layer and a system and a method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |