CN114398207A - Structured data management system, method, computer equipment and medium - Google Patents

Structured data management system, method, computer equipment and medium Download PDF

Info

Publication number
CN114398207A
CN114398207A CN202210012681.8A CN202210012681A CN114398207A CN 114398207 A CN114398207 A CN 114398207A CN 202210012681 A CN202210012681 A CN 202210012681A CN 114398207 A CN114398207 A CN 114398207A
Authority
CN
China
Prior art keywords
data
module
volume
service
disaster recovery
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210012681.8A
Other languages
Chinese (zh)
Inventor
黄开德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Eisoo Information Technology Co Ltd
Original Assignee
Shanghai Eisoo Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Eisoo Information Technology Co Ltd filed Critical Shanghai Eisoo Information Technology Co Ltd
Priority to CN202210012681.8A priority Critical patent/CN114398207A/en
Publication of CN114398207A publication Critical patent/CN114398207A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a structured data management system, a method, computer equipment and a medium. The system comprises a production business module, a data copying module, a storage module, a disaster recovery module and a recovery module; the data copying module is used for copying the business data stored in the green industry business module in real time and transmitting the business data to the disaster recovery module; the disaster recovery module is used for receiving the service data transmitted by the data replication module in real time and persisting the service data to a volume; the storage module is used for storing backup data of the business data stored in the green business module, dividing the storage space into volumes and mapping the volumes to the disaster recovery module; the disaster recovery module is used for mounting the volume; the recovery module is used for mounting the volume mapped by the storage module and recovering the backup data in the volume. The system can manage data in multiple aspects, and achieve minute-level data backup and minute-level data recovery.

Description

Structured data management system, method, computer equipment and medium
Technical Field
The embodiment of the invention relates to the technical field of structured data, in particular to a structured data management system, a method, computer equipment and a medium.
Background
In the digital age, data is a core asset of an enterprise, but with the increase of business and the explosive increase of data volume, how to effectively protect mass data becomes a difficult problem which puzzles IT construction of the enterprise.
In the existing technology based on timing backup, the backup principle is to backup data to a disaster recovery storage through periodic full-volume copy and incremental copy, and copy the data in the disaster recovery storage to a production system during recovery, thereby achieving the purpose of backup and recovery. However, in the process of increasing the data volume, the backup recovery time is also increased, and the data protection requirements of enterprises cannot be met.
Disclosure of Invention
Embodiments of the present invention provide a structured data management system, a structured data management method, a computer device, and a medium, which can perform multi-aspect management on data, and implement minute-level data backup and minute-level data recovery.
In a first aspect, an embodiment of the present invention provides a structured data management system, which includes a production service module, a data replication module, a storage module, a disaster recovery module, and a recovery module,
the production business module is respectively connected with the data copying module and the storage module, the copying module is connected with the disaster recovery module, and the storage module is connected with the recovery module;
the data copying module is used for copying the service data stored in the production service module in real time and transmitting the service data to the disaster recovery module;
the disaster recovery module is used for receiving the service data transmitted by the data replication module in real time and persisting the service data to a volume;
the storage module is used for storing backup data of the service data stored in the production service module, dividing a storage space into volumes and mapping the volumes to the disaster recovery module;
the disaster recovery module is used for mounting the volume;
and the recovery module is used for mounting the volume mapped by the storage module and recovering the backup data in the volume.
In a second aspect, an embodiment of the present invention further provides a structured data management method, including:
copying the service data stored in the production service module in real time through a copying module, and transmitting the service data to a disaster recovery module;
receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume;
storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module;
mounting the volume through the disaster recovery module;
and mounting the volume mapped by the storage module through the recovery module, and recovering the backup data in the volume.
In a third aspect, an embodiment of the present invention further provides a computer device, including:
one or more processors;
storage means for storing one or more programs;
the one or more programs are executed by the one or more processors such that the one or more processors are operable to implement the structured data management method of any embodiment of the present invention.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the structured data management method provided in any embodiment of the present invention.
The embodiment of the invention provides a structured data management system, a method, computer equipment and a medium, wherein business data stored in a production business module is copied in real time through a data copying module, and the business data is transmitted to a disaster recovery module; receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume; storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module; mounting the volume through the disaster recovery module; and mounting the volume mapped by the storage module through the recovery module, and recovering the data in the volume. By utilizing the technical scheme, the data can be managed in multiple aspects, and minute-level data backup and minute-level data recovery are realized.
Drawings
Fig. 1 is a schematic structural diagram of a structured data management system according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a structured data management system according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a structured data management system according to an exemplary embodiment of the present invention;
fig. 4 is a flowchart illustrating a structured data management method according to a third embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for managing structured data according to a fourth embodiment of the present invention;
fig. 6 is a flowchart illustrating mount recovery in a structured data management method according to a fourth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and the embodiments of the present invention are illustrative only and are not intended to limit the scope of the present invention.
It should be understood that the various steps recited in the method embodiments of the present invention may be performed in a different order and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the invention is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present invention are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in the present invention are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that reference to "one or more" unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present invention are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Example one
Fig. 1 is a schematic structural diagram of a structured data management system according to an embodiment of the present invention, where the system is applicable to a case of managing structured data, and the system may be implemented by software and/or hardware and is generally integrated on a computer device. The system may be a software system for structured data management.
As shown in fig. 1, a structured data management system according to an embodiment of the present invention includes a production business module 110, a data replication module 120, a storage module 130, a disaster recovery module 140, and a recovery module 150,
the production business module 110 is respectively connected with the data replication module 120 and the storage module 130, the replication module 120 is connected with the disaster recovery module 140, and the storage module 130 is connected with the recovery module 150;
the data replication module 120 is configured to replicate, in real time, the service data stored in the production service module, and transmit the service data to the disaster recovery module 140;
the disaster recovery module 140 is configured to receive the service data transmitted by the data replication module 120 in real time, and persist the service data to a volume;
a storage module 130, configured to store backup data of service data stored in the green business module 110, divide a storage space into volumes, and map the volumes to the disaster recovery module 140;
a disaster recovery module 140, configured to mount the volume;
and the restoring module 150 is configured to mount the volume mapped by the storage module 130, and restore the backup data in the volume.
The production service module 110 may be a module for storing service data of a user, the production service module 110 may include a service database of a plurality of production servers, the service database may store service data of a plurality of users, and the service data may be a structured data. The production services module 110 can be used to support service access.
Optionally, the production service module 110, the data replication module 120 and the storage module 130 may be connected via an ethernet, and the transmission protocol used may be, for example, a TCP/IP communication protocol.
In this embodiment, the production business module 110 is connected to the data replication module 120, and can transmit the business data that needs to be managed to the data replication module 120, so that the data replication module replicates the data, and can transmit the business data to the disaster recovery module 140 through the data replication module 120 for disaster recovery.
In this embodiment, the production business module 110, by being connected to the storage module 130, may transmit the backup data of the business data to the storage module 130, so that the storage module 130 stores the backup data.
The data replication module 120 may be a module corresponding to a data replication platform, the data replication module 120 may be a basic platform for data replication, and the data replication module 120 may be configured to perform real-time replication on service data.
In this embodiment, the data copying module 120 may be connected to the production service module 110, and configured to copy the service data transmitted by the production service module 110 in real time; the data replication module 120 may be connected to the disaster recovery module 140, and configured to replicate the service data to the disaster recovery module 140 in real time through a real-time replication technology, so as to perform real-time disaster recovery related services.
The storage module 130 may be a module corresponding to a storage medium for storing backup data, and the storage module 130 may be configured to store the backup data, divide a storage into volumes, and perform a periodic snapshot on the volumes.
Optionally, the service data may be backed up to the backup medium by the backup software through a timed backup or a copy data management technique, so as to form backup data and archive data.
Optionally, the storage may be divided into volumes of different sizes, and the size of the volume may be set arbitrarily, which is not limited herein.
Optionally, the storage module 130, the production service module 110 and the disaster recovery module 140 may be connected via an ethernet, and the transmission protocol used may be, for example, a TCP/IP communication protocol.
In this embodiment, the storage module 130 may be connected to the production service module 110, and configured to store backup data of the service data; the storage module 130 may be connected to the disaster recovery module 140, and is configured to map the volume into the disaster recovery module 140, and perform periodic snapshot on the volume in the disaster recovery module 140; the storage module 130 may also be connected to the recovery module 150 for mapping the volume to the recovery module 150, so that the recovery module 150 mounts and recovers the volume.
Further, the storage module 130 is also configured to map the volume to the production service module 110, so that the production service module 110 uses the data contained in the volume.
In this embodiment, the storage module 130 may also map the volume to a production server specified in the business module 110 so that the production server can use the volume.
Further, the storage module 130 is further configured to perform a periodic snapshot on the volume mounted in the disaster recovery module 140, and store the periodic snapshot.
It is to be understood that periodic snapshot refers to the process of snapshotting a volume at user-configured periodic intervals through a snapshot technique. After the volumes are subjected to periodic snapshot, the historical versions of the data in the disaster recovery site can be obtained, the effect of classifying the TB-level mass data in minutes is achieved, and meanwhile the requirements of users on historical data recovery can be met.
The disaster recovery module 140 may be a module corresponding to a disaster recovery site, where the disaster recovery site is a disaster recovery environment in a data disaster recovery site constructed by an enterprise, and is used to construct a master/standby environment and a read/write separation service system, where the master/standby environment and the read/write separation service system include upper layer data software, a bottom layer operating system, and storage.
Optionally, the disaster recovery module 140, the data replication platform 120 and the storage module 130 may be connected via an ethernet, and the adopted transmission protocol may be, for example, a TCP/IP communication protocol.
In this embodiment, the disaster recovery module 140 may be connected to the data replication platform 120, and configured to receive the data transmitted by the data replication module 120 in real time and persist the data to a volume; the disaster recovery module 140 may further be connected to the storage module 130, and configured to receive the volume mapped by the storage module 130, mount the volume to a server in the disaster recovery site, and support an upper database for data storage.
The recovery module 150 may be a module corresponding to a recovery site, where the recovery site may refer to an entirety of equipment, a system, and database software required in a data recovery operation, and is used to recover backup data or archived data through a mount recovery operation or a browse recovery operation.
It can be understood that mount recovery refers to that backup data and a volume where archived data is located are virtualized into lun devices through backup software and mounted into a server in a recovery site, so that a database instance can be directly started through the mounted data on lun without copying data, and a quick recovery effect is achieved.
Optionally, the connection between the recovery module 150 and the storage module 130 may be through an ethernet, and the adopted transmission protocol may be, for example, a TCP/IP communication protocol.
In this embodiment, the recovery module 150 may be connected to the storage module 130, and is configured to mount the volume mapped by the storage module 130 to a server in a recovery site, so as to support an upper database and store process data, and also be configured to load backup data in the volume, and directly pull up a database for data recovery in the recovery site, so as to provide service access to the outside.
The structured data management system provided by the first embodiment of the present invention is configured to, first, copy, in real time, service data stored in the production service module through a data copy module, and transmit the service data to the disaster recovery module; secondly, the data copying module is used for copying the service data stored in the production service module in real time and transmitting the service data to the disaster recovery module; then, the disaster recovery module is used for receiving the service data transmitted by the data replication module in real time and persisting the service data to a volume; then, the storage module is used for storing backup data of the service data stored in the production service module, dividing a storage space into volumes and mapping the volumes to the disaster recovery module; the disaster recovery module is used for mounting the volume; and the final recovery module is used for mounting the volume mapped by the storage module and recovering the backup data in the volume. By using the system, the data can be managed in multiple aspects through the production business module, the data copying module, the storage module, the disaster recovery module and the recovery module, so that minute-level data backup and minute-level data recovery are realized.
Example two
Fig. 2 is a schematic structural diagram of a structured data management system according to a second embodiment of the present invention, and the second embodiment is optimized based on the foregoing embodiments. Please refer to the first embodiment for a detailed description of the present embodiment.
As shown in fig. 2, the structured data management system provided in the second embodiment of the present invention further includes a big data module 160 on the basis of the structured data management system provided in the first embodiment, where the big data module 160 is connected to the data replication module 120, and the big data module 160 is configured to perform data integration on the service data transmitted by the data replication module 120, and use the service data in multiple services.
It can be understood that data integration refers to a process of integrating business data scattered in an enterprise into a data set in a large data platform through a data aggregation technology.
The big data module 160 may be a module corresponding to a big data platform, and for example, the big data module 160 may support services such as big data analysis and artificial intelligence.
In this embodiment, the big data module 160 may be connected to the data replication module 120 through an ethernet, and is configured to perform data integration on the service data transmitted by the big data replication module 120, so as to be used for multiple service support of a big data platform.
Further, the data copying module 120 includes a data acquisition submodule 121, a data processing submodule 122, and a data transmission submodule 123, where the data acquisition submodule 121 is configured to perform full extraction and incremental extraction on the business data stored in the industry business module 110; the data processing submodule 122 is configured to process the extracted data, where the processing includes data persistence, data encryption, data compression, concurrent transmission, and breakpoint transmission; and the data transmission submodule 123 is configured to transmit the extracted data.
The full-volume extraction and the incremental extraction refer to acquiring the full-volume data and the incremental data in the production business module 110 through a data acquisition component in the data acquisition submodule 121. The data acquisition submodule 121 may also be configured to provide breakpoint propagation.
The extracted data may refer to full data or incremental data.
The data processing sub-module 122 may also be used for data loading, data type conversion, data consistency guarantee, and the like.
The data transmission sub-module 123 may be configured to transmit the extracted full data or incremental data to the production service module 110 and the disaster recovery module 140, and may be configured to perform breakpoint transmission.
Further, the storage module 130 includes a data copy management submodule 131, a volume management submodule 132, a time management submodule 133, and a snapshot submodule 134, where the data copy management submodule 131 is configured to manage a virtual copy of the service data, and the management includes generating the virtual copy, mapping the virtual copy, mounting the virtual copy, unloading the virtual copy, and recovering the virtual copy; the volume management sub-module 132 is configured to divide the storage space into volumes of preset sizes, and perform mapping management on the volumes; a time management submodule 133, configured to manage time point information and manage all data through a timestamp, where the time point information includes a generation time of the periodic snapshot; and the snapshot submodule 134 is configured to store snapshot information of the periodic snapshot.
The data copy management submodule 131 may manage the virtual copy cloned by the virtual cloning technology based on the time point, and map the virtual copy to the production service module 110, so that the service system may access the backup data on the virtual copy, thereby achieving the effect of fast data recovery.
Optionally, the data copy management sub-module 131 may be used for, but not limited to, production of a virtual copy, mapping of a virtual copy, mount of a virtual copy, uninstall of a virtual copy, and recycle of a virtual copy.
After the volume management sub-module 132 divides the storage space into volumes of a preset size, the volumes may be mapped to a designated server in the production service module 110 or the disaster recovery module 140, and the server usage right is granted; the volume management submodule 132 may also be used for mapping management, which may include mapping, unmapping, and rewinding volumes generated, etc.
The time management submodule 133 is configured to manage all backup data and archive data by using a timestamp, and may retrieve data that needs to be restored based on the timestamp when the data is restored.
Further, the disaster recovery module 140 includes a disaster recovery database 141, where the disaster recovery database 141 is configured to receive and store the service data transmitted by the data replication module 120 in real time; the restore module 150 includes a data restore database 151, and the data restore database 151 is used to load the backup data in the volume and store the backup data.
The data recovery database is understood to be a database for data recovery.
The structured data management system provided by the second embodiment of the invention can provide data integration capability through the big data module; the data replication-based module can provide data acquisition, data transmission and data replication functions; based on the storage module, the system can provide time point management, volume management and copy management, and provide functions of periodic snapshot and the like; data recovery may be implemented based on a recovery module.
An embodiment of the present invention provides an exemplary embodiment based on the technical solutions of the foregoing embodiments, and fig. 3 is a schematic structural diagram of a structured data management system according to the exemplary embodiment of the present invention, and as shown in fig. 3, the system includes a production service module, i.e., a production service module, a silk road replication platform, i.e., a data replication module, a big data platform, i.e., a big data module, a disaster recovery site, i.e., a disaster recovery module, a snapshot storage system, i.e., a storage module, and a recovery site, i.e., a recovery module.
The production service refers to the service environment of the user, and the DB1 and the DB2 are used as service databases for storing the service data of the user and supporting service access. The data in the DB1 is used for incremental extraction and full extraction of the SilkRoad replication platform, and the data in the DB2 is used for transmitting the backup data to the snapshot storage system after backup.
The SilkRoad replication platform is used as a basic platform for data replication, is used for supporting the real-time replication of production service data, can perform full extraction and incremental extraction on a service database in the production service, and transmits and loads the service database to a big data platform in real time for data integration and transmits the service database to a disaster recovery site for real-time disaster recovery related services.
The SilkRoad replication platform has a data acquisition function, a data transmission function and a data loading function. The data acquisition function is used for carrying out full data extraction and incremental data extraction on a database in the production service and providing breakpoint continuous transmission capability; the data transmission function is used for transmitting the acquired data, and provides but is not limited to data persistence, data encryption, data compression, concurrent transmission and breakpoint continuous transmission capabilities; the data loading function is used for loading data to a big data platform and a disaster recovery site, and provides but is not limited to data loading, data type conversion, data consistency guarantee and breakpoint continuous transmission capability.
The big data platform is used for integrating production service data and converging data so as to support services such as big data analysis and artificial intelligence, and can comprise Hadoop and other open source or commercialized BigData platforms.
The disaster recovery site is a disaster recovery environment in a data disaster backup site constructed by an enterprise, is used for constructing a main and standby environment, and is a read-write separation business system, and comprises upper layer data software, a bottom layer operating system and storage.
The Lun1 in the disaster recovery site represents a volume mapped by the snapshot storage system, and is mounted in a server in the disaster recovery site to support an upper database for data storage. The DB3 in the disaster recovery site represents a disaster recovery database in the disaster recovery site for receiving data transmitted by the production service, which can receive the transmitted data in real time and persist the data to the storage Lun 1.
The snapshot storage system is a storage medium used for storing backup data in the disaster backup system, can store data which is backed up based on timing backup and copy data management, can also divide a volume for storage, and maps the volume to a service server for use, and carries out periodic snapshot on the volume.
The snapshot storage system has a volume management function, a time point management function and a data copy management function. The storage can be divided into volumes with specified size through volume management, and the volumes are mapped to a specified server, the server is granted with use right, and the volumes generated are managed and mapped, and the volumes are unmapped and returned, including but not limited to the above functions; point-in-time management may be understood as managing all backup data and archive data by means of a timestamp, and at the time of restoration, a data set to be restored may be retrieved based on the timestamp; the data copy management function refers to management of virtual copies cloned by a virtual cloning technology based on time points, and the virtual copies are mapped to a service server, so that a service system can access backup data on the virtual copies, and a quick recovery effect is achieved.
The recovery site refers to a whole of machine equipment, a system and database software which need to be prepared when data recovery operation is performed, and is used for recovering backup data or archived data to the environment through mount recovery or browse recovery operation.
The Lun2 in the recovery site represents the volume mapped by the snapshot storage system, and is mounted in a server in the recovery site for supporting an upper database for data storage. DB4 in the recovery site represents a database for data recovery in the recovery site, and can load backup data mounted by Lun2 and directly pull up the database to provide service access to the outside.
The structured data management system provided by the embodiment of the invention can realize the capability of converging the dispersed data in an enterprise and integrating the data into a large data platform; the internal data of the enterprise can be backed up at regular time, the copy data management and other capabilities are realized, and the data protection and filing requirements are met; the method can be used for disaster tolerance and migration of data in the enterprise, and meets the requirements of high availability, read-write separation, data migration and the like of enterprise services; the internal data of the enterprise can be backed up in real time, the requirements of clients for real-time backup of TB-level mass data are met, and the risk of data loss is effectively reduced; the backup data can be mounted and restored, and the effect of quick restoration of the service is met; the disaster recovery backup data can be distributed, the user data utilization popularity is met, the requirements of development and testing and big data analysis are met, and the value of the disaster recovery backup data is explored.
EXAMPLE III
Fig. 4 is a flowchart illustrating a structured data management method according to a third embodiment of the present invention, where the method is applicable to a case of managing structured data, and the method can be executed by a computer device, where the computer device includes a structured data management system according to any embodiment of the present invention.
It should be noted that the structured data management method needs to be executed when the following conditions are satisfied, and the conditions may include:
1. it is necessary to provide access rights to the production service, including connection, inquiry, etc.
2. The access rights including connection, inquiry and other rights of the disaster recovery database in the disaster recovery site need to be provided.
3. Access rights including connection, query, etc. for recovering the database in the recovery site need to be provided.
4. Transaction log access for production services is provided so that it can read database transaction logs and parse transaction records therein.
5. The operating system type, version of database software, and bit requirements in the recovery environment of the recovery site are consistent with those in the production business environment.
6. The production service environment and the recovery environment need to deploy a client of backup software for executing backup recovery operation and data extraction operation.
7. And during mounting recovery, ten-trillion network connection or FC network connection is needed between the recovery environment and the disaster recovery platform device.
As shown in fig. 4, a structured data management method provided by the third embodiment of the present invention includes the following steps:
s110, copying the service data stored in the production service module in real time through the data copying module, and transmitting the service data to the disaster recovery module.
The production service module may correspond to a plurality of production servers, a service database in the production server may store service data of a plurality of users, and the service data may be a kind of structured data.
And S120, receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume.
The data replication module can be used as a basic platform for data replication, and the data replication module can be used for performing real-time replication on service data.
S130, storing the backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module.
The backup data can be stored through the storage module, the volume can be divided for storage, and the volume can be subjected to periodic snapshot.
S140, mounting the volume through the disaster recovery module.
The data transmitted by the data replication module can be received in real time through the disaster recovery module and persisted to the volume, and the volume mapped by the storage module can be received through the disaster recovery module and mounted in a server in a disaster recovery site to support an upper database for data storage.
S150, mounting the volume mapped by the storage module through the recovery module, and recovering the data in the volume.
The backup data or the archived data can be restored through mounting restoration or browsing restoration operation through the restoration module.
Firstly, copying the service data stored in the production service module in real time through a data copying module, and transmitting the service data to a disaster recovery module; secondly, receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume; then, storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module; then the volume is mounted through the disaster recovery module; and finally, mounting the volume mapped by the storage module through the recovery module, and recovering the data in the volume. By the method, data can be managed in multiple aspects, and minute-level data backup and minute-level data recovery are realized.
Further, the method further comprises: and performing data integration on the service data transmitted by the data copying module through the big data module, and using the service data for multiple services.
Further, the method further comprises: mapping, by a storage module, the volume to the production business module to cause the production business module to use the data contained in the volume.
Further, the method further comprises: carrying out periodic snapshot on the volume mounted in the disaster recovery module through a storage module, and storing the periodic snapshot;
further, the method further comprises: performing full extraction and incremental extraction on the service data stored by the production service module through a data acquisition submodule in a data copying module; processing the extracted data through a data processing submodule in the data copying module, wherein the processing comprises data persistence, data encryption, data compression and breakpoint continuous transmission; and transmitting the extracted data through a data transmission submodule in a data copying module.
Further, the method further comprises: managing the virtual copy of the service data through a data copy management submodule in a storage module, wherein the management comprises generating the virtual copy, mapping the virtual copy, mounting the virtual copy, unloading the virtual copy and recovering the virtual copy; dividing the storage space into volumes with preset sizes through a volume management submodule in a storage module, and carrying out mapping management on the volumes; managing time point information and managing all data through a timestamp by a time management submodule in a storage module, wherein the time point information comprises the generation time of the periodic snapshot; and storing the snapshot information of the periodic snapshot through a snapshot submodule in a storage module.
Further, the method further comprises: receiving and storing the service data transmitted by the data copying module in real time through a disaster tolerance database in a disaster tolerance module; and loading the backup data in the volume through a data recovery database in the disaster recovery module, and storing the backup data.
Example four
Fig. 5 is an exemplary flowchart of a structured data management method according to a fourth embodiment of the present invention, where the flowchart in fig. 5 only relates to a part of modules in the structured data management system. By the method, the data disaster tolerance task can be executed. As shown in fig. 5, the method may include the following processes:
step 1, after an operator logs in a system, a data disaster tolerance task is established.
And 2, selecting service data needing to be protected in the production service on a system page, and confirming that a database in the production service is in a running state.
And 3, selecting a disaster recovery server in the disaster recovery site on the system page, confirming that the database software is installed on the disaster recovery server, and normally operating.
And 4, creating a volume through a snapshot storage system in the structured data management system, mapping the volume into the disaster recovery site, and mounting the volume into the disaster recovery server.
And 5, creating a disaster recovery database on the disaster recovery server, and pointing the data storage position to the mounted volume.
And 6, creating a data replication link between the server and the SilkRoad replication platform in the production service, and verifying the availability of the link.
And 7, judging whether a data disaster tolerance task needs to be started by an operator.
If yes, executing step 8; if not, go to step 16.
And 8, collecting the total data in the production service through the SilkRoad replication platform, and synchronizing the data to the disaster recovery database.
And 9, judging whether the volume mounted on the disaster recovery server needs to be subjected to snapshot through a snapshot storage system.
If yes, executing step 10; if not, go to step 13.
And step 10, carrying out consistency processing on the disaster tolerance database, and temporarily freezing the disaster tolerance database.
And step 11, carrying out snapshot on the volume mounted by the disaster recovery server through a snapshot storage system, and unfreezing the frozen disaster recovery database.
And step 12, exemplifying time point information and snapshot information by a snapshot storage system.
And step 13, acquiring incremental data in the production service through the SilkRoad replication platform, and synchronizing the incremental data to the disaster recovery database.
And step 14, judging whether the data disaster tolerance task needs to be stopped by an operator.
If yes, go to step 15; if not, go to step 9.
And 15, stopping the data disaster tolerance task.
The data disaster tolerance task can be executed through a production business module, a data copying module, a storage module and a disaster tolerance module in the structured data management system, and the data disaster tolerance can be realized.
Fig. 6 is a mount recovery flow chart in a structured data management method according to a fourth embodiment of the present invention, and as shown in fig. 6, a mount recovery process includes the following flows:
step 1, an operator creates a mount recovery task.
And 2, selecting the database needing to be restored as a restoration database according to the metadata information of the backup data.
And 3, selecting a time point needing to be recovered according to the recovery requirement.
And 4, selecting a server needing to be recovered in the recovery site.
And 5, retrieving the corresponding snapshot according to the time point selected by the user, and creating a virtual copy.
And 6, mapping the volume corresponding to the virtual copy to a recovery server in the recovery site by the snapshot storage system.
And 7, mounting the mapped volume into a recovery server.
And 8, restoring database scanning and loading data on the volume.
And 9, unfreezing through database software to recover the database, and starting a database instance.
And step 10, verifying the availability of the recovery database and providing service access to the outside.
The mount recovery process can be executed in a recovery module in the structured data management system, and the recovery of data can be realized.
EXAMPLE five
Fig. 7 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention. As shown in fig. 7, a computer device according to a fifth embodiment of the present invention includes: one or more processors 41 and storage 42; the processor 41 in the computer device may be one or more, and fig. 7 illustrates one processor 41 as an example; storage 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41, such that the one or more processors 41 implement the structured data management method according to any of the embodiments of the present invention.
The computer device may further include: an input device 43 and an output device 44.
The processor 41, the storage device 42, the input device 43 and the output device 44 in the computer apparatus may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 7.
The storage device 42 in the computer apparatus is used as a computer-readable storage medium for storing one or more programs, which may be software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the structured data management method provided in the third embodiment of the present invention (for example, the modules in the structured data management system shown in fig. 1, including the production business module 110, the data replication module 120, the disaster recovery module 130, the storage module 140, and the recovery module 150). The processor 41 executes various functional applications and data processing of the computer device by executing software programs, instructions and modules stored in the storage device 42, namely, implements the structured data management method in the above method embodiment.
The storage device 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the storage 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 42 may further include memory located remotely from processor 41, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 43 may be used to receive input numeric or character information and to generate key signal inputs related to user settings and function controls of the computer apparatus. The output device 44 may include a display device such as a display screen.
And, when one or more programs included in the above-described computer apparatus are executed by the one or more processors 41, the programs perform the following operations:
the method comprises the steps that business data stored in a production business module are copied in real time through a data copying module, and the business data are transmitted to a disaster recovery module;
receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume;
storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module;
mounting the volume through the disaster recovery module;
and mounting the volume mapped by the storage module through the recovery module, and recovering the data in the volume.
EXAMPLE six
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is used, when executed by a processor, to execute a structured data management method, where the method includes:
the method comprises the steps that business data stored in a production business module are copied in real time through a data copying module, and the business data are transmitted to a disaster recovery module;
receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume;
storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module;
mounting the volume through the disaster recovery module;
and mounting the volume mapped by the storage module through the recovery module, and recovering the data in the volume.
Optionally, the program, when executed by a processor, may be further configured to perform a method for structured data management according to any of the embodiments of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A structured data management system is characterized in that the system comprises a production business module, a data copying module, a storage module, a disaster recovery module and a recovery module,
the production business module is respectively connected with the data copying module and the storage module, the copying module is connected with the disaster recovery module, and the storage module is connected with the recovery module;
the data copying module is used for copying the service data stored in the production service module in real time and transmitting the service data to the disaster recovery module;
the disaster recovery module is used for receiving the service data transmitted by the data replication module in real time and persisting the service data to a volume;
the storage module is used for storing backup data of the service data stored in the production service module, dividing a storage space into volumes and mapping the volumes to the disaster recovery module;
the disaster recovery module is used for mounting the volume;
and the recovery module is used for mounting the volume mapped by the storage module and recovering the backup data in the volume.
2. The system of claim 1, further comprising a big data module coupled to the data replication module,
and the big data module is used for performing data integration on the service data transmitted by the data copying module and using the service data for multiple services.
3. The system of claim 1, wherein the storage module is further configured to map the volume to the production business module, such that the production business module uses the data contained in the volume.
4. The system according to claim 1, wherein the storage module is further configured to perform a periodic snapshot on the volume mounted in the disaster recovery module, and store the periodic snapshot.
5. The system of claim 1, wherein the data replication module comprises a data acquisition sub-module, a data processing sub-module, and a data transmission sub-module,
the data acquisition submodule is used for performing full extraction and incremental extraction on the service data stored by the production service module;
the data processing submodule is used for processing the extracted data, and the processing comprises data persistence, data encryption, data compression and breakpoint continuous transmission;
and the data transmission submodule is used for transmitting the extracted data.
6. The system of claim 1, wherein the storage module comprises a data copy management submodule, a volume management submodule, a time management submodule, and a snapshot submodule,
the data copy management submodule is used for managing the virtual copy of the service data, and the management comprises the generation of the virtual copy, the mapping of the virtual copy, the mounting of the virtual copy, the unloading of the virtual copy and the recovery of the virtual copy;
the volume management submodule is used for dividing the storage space into volumes with preset sizes and carrying out mapping management on the volumes;
the time management submodule is used for managing time point information and managing all data through a timestamp, and the time point information comprises the generation time of the periodic snapshot;
and the snapshot submodule is used for storing snapshot information of the periodic snapshot.
7. The system of claim 1,
the disaster recovery module comprises a disaster recovery database, and the disaster recovery database is used for receiving and storing the service data transmitted by the data replication module in real time;
the recovery module includes a data recovery database for loading the backup data in the volume and storing the backup data.
8. A method of structured data management, the method being performed by the structured data management system of claim 1, the method comprising:
the method comprises the steps that business data stored in a production business module are copied in real time through a data copying module, and the business data are transmitted to a disaster recovery module;
receiving the service data transmitted by the data copying module in real time through a disaster recovery module, and persisting the service data to a volume;
storing backup data of the service data stored in the production service module through a storage module, dividing a storage space into volumes, and mapping the volumes to the disaster recovery module;
mounting the volume through the disaster recovery module;
and mounting the volume mapped by the storage module through the recovery module, and recovering the backup data in the volume.
9. A computer device, comprising:
one or more processors;
storage means for storing one or more programs;
the one or more programs being executable by the one or more processors to cause the one or more processors to perform the structured data management method of claim 8.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the structured data management method of claim 8.
CN202210012681.8A 2022-01-07 2022-01-07 Structured data management system, method, computer equipment and medium Pending CN114398207A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210012681.8A CN114398207A (en) 2022-01-07 2022-01-07 Structured data management system, method, computer equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210012681.8A CN114398207A (en) 2022-01-07 2022-01-07 Structured data management system, method, computer equipment and medium

Publications (1)

Publication Number Publication Date
CN114398207A true CN114398207A (en) 2022-04-26

Family

ID=81229916

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210012681.8A Pending CN114398207A (en) 2022-01-07 2022-01-07 Structured data management system, method, computer equipment and medium

Country Status (1)

Country Link
CN (1) CN114398207A (en)

Similar Documents

Publication Publication Date Title
US10846266B2 (en) Distributed architecture for content indexing emails
US20210208972A1 (en) Application aware backup of virtual machines
US11288135B2 (en) Synthesizing a restore image from one or more secondary copies to facilitate data restore operations to a file server
US20200401485A1 (en) Using a pseudo-disk driver for indexing contents and metadata of block-level backup copies, including block-level backup copies of virtual machine data files
US11687511B2 (en) Distributed framework for data proximity-based task splitting in a content indexing system
US20180285201A1 (en) Backup operations for large databases using live synchronization
US11321190B2 (en) Distributed framework for task splitting and task assignments in a content indexing system
US20200073574A1 (en) Data migration using stubs
US11431797B2 (en) System for managing multiple information management cells
US20210334171A1 (en) Distributed content indexing architecture with separately stored file previews
US11609827B2 (en) Distributed architecture for tracking content indexing
US20070294310A1 (en) Method and apparatus for storing and recovering fixed content
US20190087285A1 (en) Distributed content indexing architecture with combined backup and content indices
US20190087286A1 (en) Distributed architecture for content indexing using restored secondary copies
US20220156190A1 (en) Re-stubbing migrated data
US11442623B2 (en) Faster browse of secondary copies of block-level data volumes
US20220188719A1 (en) Systems and methods for generating a user file activity audit report
US20220318199A1 (en) Seamless migration of stubs and data between different filer types
WO2014133502A1 (en) Sending a request to a management service
US20230153010A1 (en) Pruning data segments stored in cloud storage to reclaim cloud storage space
CN114398207A (en) Structured data management system, method, computer equipment and medium
US11474753B2 (en) Systems and methods for backing up to a virtual tape library
US20220043863A1 (en) Propagating filters to multiple dashboard user interfaces in an information management environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination