WO2021208402A1 - 现代应用的副本数据管理系统及方法 - Google Patents

现代应用的副本数据管理系统及方法 Download PDF

Info

Publication number
WO2021208402A1
WO2021208402A1 PCT/CN2020/124230 CN2020124230W WO2021208402A1 WO 2021208402 A1 WO2021208402 A1 WO 2021208402A1 CN 2020124230 W CN2020124230 W CN 2020124230W WO 2021208402 A1 WO2021208402 A1 WO 2021208402A1
Authority
WO
WIPO (PCT)
Prior art keywords
metadata
object storage
database
production
copy
Prior art date
Application number
PCT/CN2020/124230
Other languages
English (en)
French (fr)
Inventor
周丰
贺鸿富
童莎
Original Assignee
上海爱数信息技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海爱数信息技术股份有限公司 filed Critical 上海爱数信息技术股份有限公司
Priority to US17/996,248 priority Critical patent/US20230214301A1/en
Publication of WO2021208402A1 publication Critical patent/WO2021208402A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1464—Management of the backup or restore process for networked environments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1415—Saving, restoring, recovering or retrying at system level
    • G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2094—Redundant storage or storage space
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Definitions

  • This application relates to object storage technology, such as a copy data management system and method for modern applications.
  • the modern application system includes business units, databases and object storage.
  • the characteristic is that the data is stored in the object storage and the metadata is stored in the database.
  • the metadata and the object storage in the database are backed up to generate duplicate data.
  • the copy data is used to restore, to ensure the safety of the data of the entire modern application system.
  • the data saved by modern application systems may be small files.
  • the modern application system is damaged, the objects in the object storage need to be restored to the reconstructed system one by one.
  • the recovery time will be very long.
  • This application provides a copy data management system and method for modern applications, which has good security and makes full use of stored data.
  • a copy data management system for modern applications including:
  • the modern application module includes a business unit, an object storage gateway, a production object storage and a database, the business unit is connected to the object storage gateway and the database, and the business unit is set to generate large objects and small objects.
  • Object and business metadata send the large object and the small object to the object storage gateway and store the business metadata in the database, the object storage gateway and the production object store and store
  • the database connection, the production object storage gateway is set to write and read large objects and small objects in the production object storage, merge small objects into large objects, generate object metadata, and store the object metadata in all
  • the database is set to store business metadata and object metadata;
  • a copy data management module includes a metadata processing unit and a disaster recovery object storage
  • the metadata processing unit is connected to the database
  • the metadata processing unit is configured to back up the database to generate metadata
  • the disaster recovery object storage is connected to the object storage gateway
  • the object storage gateway is further configured to back up the production object storage to the disaster recovery object storage;
  • the metadata processing unit is further configured to use the metadata copy and its snapshot to realize the restoration of modern application modules and the utilization of the copy;
  • the metadata processing unit performs snapshots and clones of metadata copies in sequence, generates clone metadata snapshots, mounts the clone metadata snapshots to the database, and reads the large objects in the disaster recovery object storage in a read-only manner through the object storage gateway .
  • the business unit is further configured to send an object write request and an object read request to the object storage gateway;
  • the object storage gateway is set to store the received large object or small object in the production object storage when the object write request is received, and the data volume of the small object in the production object storage reaches the set threshold In this case, merge small objects into large objects; in the case of receiving an object read request, detect whether there is object metadata in the database, and in response to the detection result that there is object metadata in the database, access the production object storage according to the object metadata In response to the detection result that there is no object metadata in the production object database, the data at the specified location and length in the large object in the large object in the production object store is directly read.
  • the restoration of the modern application module includes damage restoration and historical restoration
  • the damage recovery process includes: the metadata processing unit restores the latest metadata copy to the database, and the object storage gateway synchronizes the production object storage with the disaster recovery object storage;
  • the history restoration process includes: the metadata processing unit restores the snapshot of the metadata copy at the specified time point to the database, the object storage gateway queries the database for the large object corresponding to the specified time point, and the large object The object is restored to the production object store.
  • the business metadata includes user information, user permissions, user file storage information, and object names
  • the object metadata includes large object names, offsets, and lengths.
  • the object storage gateway adopts the object storage S3 interface.
  • a copy data management method for modern applications including:
  • the business unit generates large objects, small objects, and business metadata, sends the large objects and the small objects to the object storage gateway, and stores the business metadata in the database;
  • the object storage gateway is in the production object Storage writes and reads large objects and small objects and merges small objects into large objects, generates object metadata, stores the object metadata in the database, and the metadata processing unit backs up the database to generate a copy of the metadata, the production object
  • the storage gateway backs up the production object storage to the disaster recovery object storage;
  • the metadata processing unit uses metadata copies and their snapshots to realize database and production object storage recovery and copy utilization
  • the copy utilization process includes:
  • the metadata processing unit performs snapshots and clones of metadata copies in sequence, generates clone metadata snapshots, mounts the clone metadata snapshots to the database, and reads the large objects in the disaster recovery object storage in a read-only manner through the object storage gateway .
  • the method also includes:
  • the business unit sends an object write request and an object read request to the object storage gateway;
  • the object storage gateway When the object storage gateway receives an object write request, the object storage gateway stores the received large object or small object in the production object storage, and the data volume of the small object in the production object storage reaches the set threshold. In this case, the object storage gateway merges small objects into large objects;
  • the object storage gateway When the object storage gateway receives the object read request, the object storage gateway detects whether the database has object metadata, and in response to the detection result that there is object metadata in the database, accesses the production object storage in accordance with the object metadata Data of the specified location and length in the large object, in response to the detection result that there is no object metadata in the database, directly read the large object in the production object storage;
  • the recovery of the production object database and production object storage includes damage recovery and historical recovery;
  • the damage recovery process includes: the metadata processing unit restores the latest metadata copy to the database, and the object storage gateway synchronizes the production object storage with the disaster recovery object storage;
  • the historical restoration includes: the metadata processing unit restores the snapshot of the metadata copy at the specified time point to the database, the object storage gateway queries the database for the large object corresponding to the specified time point, and the large object Revert to production object storage.
  • the business metadata includes user information, user permissions, user file storage information, and object names
  • the object metadata includes large object names, offsets, and lengths.
  • the object storage gateway adopts the object storage S3 interface.
  • FIG. 1 is a schematic diagram of the framework of a copy data management system of a modern application provided by an embodiment of the application.
  • This embodiment provides a copy data management system for modern applications.
  • the copy data management system for modern applications provided in this embodiment includes a modern application module and a copy data management module.
  • Modern application modules include business units, object storage gateways, production object storage and databases.
  • the business units are set to generate large and small objects and business metadata.
  • the object storage gateway is set to write and read large objects and small objects in the production object storage.
  • Object merge small objects into large objects and generate object metadata
  • the database is set to store business metadata and object metadata.
  • the object metadata includes the name, offset, and length of the large object.
  • the offset is the starting position of the small object in the large object, and the length includes the length of the small object.
  • Business metadata is the data required by the business unit to process the business.
  • Business metadata is the data processed by the business unit, including user information, user permissions, user file storage information, and object names.
  • the object storage gateway adopts the object storage S3 interface, and the address for accessing production object storage and disaster recovery object storage is changed to the address of the object storage gateway, so that all operations on production object storage and disaster recovery object storage are taken over by the object storage gateway.
  • the copy data management module includes a metadata processing unit and disaster recovery object storage.
  • the metadata processing unit is set to back up metadata, generate and store metadata copies
  • the disaster recovery object storage is set to back up production object storage to protect production. The purpose of object storage.
  • the metadata processing unit uses the metadata copy and its snapshot to realize the restoration of modern application modules and the utilization of the copy.
  • the copy utilization process includes:
  • the metadata processing unit performs snapshots and clones of metadata copies in sequence, generates clone metadata snapshots, mounts the clone metadata snapshots to the database, and reads the large objects in the disaster recovery object storage in a read-only manner through the object storage gateway , To provide data for third-party data analysis, data mining, development and testing systems, which not only ensures the safety of production object storage, but also makes full use of the data stored in production objects.
  • the business unit is also set to send object write requests and object read requests to the object storage gateway; when the object storage gateway receives the object write request, the object storage gateway stores the large and small objects in the production object storage. When the amount of object data reaches the set threshold, the object storage gateway merges small objects into large objects; when the object storage gateway receives an object read request, it detects whether there is object metadata in the database. If there is object metadata in the database, it will Metadata or metadata copy accesses data at a specified location and length in a large object. If there is no object metadata in the database, the large object is directly read.
  • the recovery of modern application modules includes damage recovery and historical recovery
  • the process includes: restoring the latest metadata copy to the database, and synchronizing the disaster recovery object storage to the production object storage;
  • the historical restoration is performed.
  • the process includes: the metadata processing unit restores the snapshot of the metadata copy to the database, and the metadata processing unit queries the corresponding large object by querying the database to restore the large object
  • the effect of restoring to the historical point in time can be achieved.
  • the implementation is as follows:
  • Deploy the object storage gateway configure the production object storage and disaster recovery object storage information to the object storage gateway; modify the addresses of the production object storage and disaster recovery object storage in the document cloud to the address of the object storage gateway; deploy the copy data management module , Back up the production object storage to the disaster recovery object storage, and back up the database to the metadata processing unit to complete data protection; clone and mount the metadata snapshot at the specified time point to the database, and access the disaster recovery object storage through the object storage gateway , Provide data for third-party data analysis, data mining, development and testing systems.
  • the large object and the small object in this embodiment respectively correspond to the large file and the small file of the business unit.
  • a large file is a file with a data volume greater than a threshold
  • a small file is a file with a data volume less than or equal to the threshold.
  • the business unit generates large and small objects and business metadata, stores the business metadata in the database, and sends object write requests and object read requests to the object storage gateway;
  • the object storage gateway When the object storage gateway receives the object write request, the large and small objects are stored in the production object storage. When the amount of small object data in the production object storage reaches the set threshold, the object storage gateway merges the small objects into large objects and generates the objects Metadata, the object metadata is stored in the database, the object storage gateway uses the object storage S3 interface, and the disaster recovery object storage backs up the production object storage.
  • the object metadata includes the large object name, offset and length, and the business metadata is the business
  • the data processed by the unit includes user information, user permissions, user file storage information and object names;
  • the object storage gateway When the object storage gateway receives an object read request, it detects whether there is object metadata in the database. If there is object metadata in the database, it accesses the data at the specified location and length in the large object according to the object metadata. If there is no object metadata in the database , Then read the large object directly;
  • the metadata processing unit uses metadata copies and their snapshots to realize database and production object storage recovery and copy utilization
  • the metadata processing unit performs snapshots and clones of metadata copies in sequence, generates clone metadata snapshots, mounts the clone metadata snapshots to the database, and reads the data in the disaster recovery object storage in a read-only manner through the object storage gateway.
  • Objects provide data for third-party data analysis, data mining, development and testing systems, which not only ensures the safety of production object storage, but also makes full use of the data stored in production objects.
  • the recovery of database and production object storage includes damage recovery and historical recovery
  • the damage recovery process includes: the metadata processing unit restores the latest metadata copy to the database, and synchronizes the production object storage with the disaster recovery object storage;
  • the historical restoration includes: the metadata processing unit restores the snapshot of the metadata copy at the specified time point to the database, the disaster recovery object storage queries the corresponding large object by querying the database, and restores the large object to the production object Store, you can achieve the effect of restoring to the historical point in time.
  • Embodiment 1 and Embodiment 2 propose a copy data management system and method for modern applications.
  • the large and small objects, business metadata and object metadata of modern application systems To protect and reuse the backup data, give full play to the value of the backup data, and implement an object storage gateway to merge small objects into large objects, so that the data in the object storage can be quickly migrated to the copy data management system.
  • an object storage gateway to merge small objects into large objects, so that the data in the object storage can be quickly migrated to the copy data management system.
  • it can recover quickly, greatly improve the performance of data protection, and solve the problem of data protection and reuse in modern applications.
  • This application integrates small objects through the object storage gateway using object metadata access, and backs up the production object storage and production object database, correspondingly obtains disaster recovery object storage and metadata copies, and performs snapshots and clones of the metadata copies in sequence. Obtain clone metadata snapshots to restore production object storage quickly. Use clone metadata snapshots to mount to the disaster recovery object database, and access the disaster recovery object storage in read-only mode through the disaster recovery object storage gateway, which can make full use of the disaster recovery objects The data stored in the storage does not affect the security of the production object storage at the same time, and the security is good.
  • This application can perform damage recovery and historical recovery on production object storage.
  • the metadata processing unit restores the latest metadata copy to the production object database, synchronizes the disaster recovery object storage to the production object storage, and performs historical recovery.
  • the snapshot of the metadata copy at the specified time point is restored to the production object database, and the disaster recovery object storage queries the corresponding large object by querying the database and restores the large object to the production object storage, which is safe and easy to operate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Library & Information Science (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种现代应用的副本数据管理系统及方法，该系统包括：现代应用模块，所述现代应用模块包括业务单元、对象存储网关、生产对象存储和数据库，所述业务单元设置为生成大对象、小对象以及业务元数据，所述对象存储网关设置为在生产对象存储中写入和读取大对象、小对象，将小对象合并为大对象并生成对象元数据，所述数据库设置为存储业务元数据和对象元数据；副本数据管理模块，所述副本数据管理模块包括元数据处理单元和灾备对象存储，所述元数据处理单元对数据库进行备份，生成元数据副本，所述生产对象存储网关设置为将所述生产对象存储备份到所述灾备对象存储；所述元数据处理单元利用元数据副本及其快照实现现代应用模块的恢复以及副本利用。

Description

现代应用的副本数据管理系统及方法
本申请要求在2020年04月14日提交中国专利局、申请号为202010290216.1的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及对象存储技术，例如涉及一种现代应用的副本数据管理系统及方法。
背景技术
随着互联网的快速发展以及云技术的落地，移动办公和跨地域协作越来越多，在这样的背景下，多种基于对象存储构建的现代应用系统应运而生并快速发展，大量的数据涌入现代应用系统，现代应用系统包括业务单元、数据库以及对象存储，特点是数据保存在对象存储中，元数据保存在数据库中，同时通过对数据库中的元数据以及对象存储进行备份，生成副本数据，当现代应用系统出现损坏需要恢复时利用副本数据进行恢复，保证整个现代应用系统的数据的安全。
但是现代应用系统保存的数据可能都是一些小文件，当现代应用系统损坏后，需要将对象存储中的对象一个个恢复到重建的系统中，在海量小对象场景下，恢复的时间会很长，达不到快速恢复系统的要求，同时这些副本数据只能在系统恢复时被利用，未能有效利用，无法发挥副本数据的利用价值。
发明内容
本申请提供一种现代应用的副本数据管理系统及方法，安全性好，充分利用存储数据。
本申请可以通过以下技术方案来实现：
一种现代应用的副本数据管理系统，包括：
现代应用模块，所述现代应用模块包括业务单元、对象存储网关、生产对象存储和数据库，所述业务单元与所述对象存储网关以及所述数据库连接，所 述业务单元设置为生成大对象、小对象以及业务元数据，将所述大对象和所述小对象发送至所述对象存储网关以及将所述业务元数据存储至所述数据库中，所述对象存储网关与所述生产对象存储和所述数据库连接，所述生产对象存储网关设置为在生产对象存储中写入和读取大对象和小对象，将小对象合并为大对象并生成对象元数据以及将所述对象元数据存储至所述数据库，所述数据库设置为存储业务元数据以及对象元数据；
副本数据管理模块，所述副本数据管理模块包括元数据处理单元和灾备对象存储，所述元数据处理单元与所述数据库连接，所述元数据处理单元设置为对数据库进行备份，生成元数据副本，所述灾备对象存储与所述对象存储网关连接，所述对象存储网关还设置为将所述生产对象存储备份到所述灾备对象存储；
所述元数据处理单元还设置为利用元数据副本及其快照实现现代应用模块的恢复以及副本利用；
所述的副本利用过程为：
所述元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾备对象存储中的大对象。
所述业务单元还设置为向对象存储网关发送对象写请求和对象读请求；
所述对象存储网关是设置为：在接收到对象写请求的情况下，将接收到的大对象或小对象存储至生产对象存储，在生产对象存储中的小对象的数据量达到设定阈值的情况下，将小对象合并为大对象；在接收到对象读请求的情况下，检测数据库中是否有对象元数据，响应于数据库中有对象元数据的检测结果，根据对象元数据访问生产对象存储中的大对象中指定位置以及长度的数据，响 应于生产对象数据库中没有对象元数据的检测结果，直接读取生产对象存储中的大对象。
所述现代应用模块的恢复包括损坏恢复和历史恢复；
所述损坏恢复过程包括：所述元数据处理单元将最新的元数据副本恢复至数据库，对象存储网关将所述生产对象存储与灾备对象存储同步；
所述历史恢复过程包括：所述元数据处理单元将指定时间点的元数据副本的快照恢复至数据库，所述对象存储网关通过查询数据库查询与指定时间点对应的大对象，并将所述大对象恢复至生产对象存储。
所述业务元数据包括用户信息、用户权限、用户文件存储信息和对象名称，所述对象元数据包括大对象名称、偏移量和长度。
所述对象存储网关采用对象存储S3接口。
一种现代应用的副本数据管理方法，包括：
业务单元生成大对象、小对象以及业务元数据，将所述大对象和所述小对象发送至所述对象存储网关以及将所述业务元数据存储至所述数据库中；对象存储网关在生产对象存储写入和读取大对象和小对象并将小对象合并为大对象，生成对象元数据，将对象元数据存储至数据库，元数据处理单元对数据库进行备份生成元数据副本，所述生产对象存储网关将所述生产对象存储备份到灾备对象存储；
元数据处理单元利用元数据副本及其快照实现数据库和生产对象存储的恢复以及副本利用；
所述副本利用过程包括：
所述元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾 备对象存储中的大对象。
所述方法还包括：
所述业务单元向对象存储网关发送对象写请求和对象读请求；
在所述对象存储网关接收到对象写请求的情况下，所述对象存储网关将接收的大对象或小对象存储至生产对象存储，在生产对象存储中的小对象的数据量达到设定阈值的情况下，对象存储网关将小对象合并为大对象；
所述对象存储网关在接收到对象读请求的情况下，所述对象存储网关检测数据库是否有对象元数据，响应于数据库中有对象元数据的检测结果，根据对象元数据访问生产对象存储中的大对象中指定位置以及长度的数据，响应于数据库中没有对象元数据的检测结果，直接读取生产对象存储中的大对象；
所述生产对象数据库和生产对象存储的恢复包括损坏恢复和历史恢复；
所述损坏恢复过程包括：所述元数据处理单元将最新的元数据副本恢复至数据库，对象存储网关将所述生产对象存储与灾备对象存储同步；
所述历史恢复包括：所述元数据处理单元将指定时间点的元数据副本的快照恢复至数据库，所述对象存储网关通过查询数据库查询与指定时间点对应的大对象，并将所述大对象恢复至生产对象存储。
所述业务元数据包括用户信息、用户权限、用户文件存储信息和对象名称，所述对象元数据包括大对象名称、偏移量和长度。
所述对象存储网关采用对象存储S3接口。
附图说明
图1为本申请实施例提供的一种现代应用的副本数据管理系统的框架示意图。
具体实施方式
下面结合附图和实施例对本申请进行说明。本实施例以本申请技术方案为前提进行实施，给出了实施方式和操作过程，但本申请的保护范围不限于下述的实施例。
实施例1
本实施例提供一种现代应用的副本数据管理系统，如图1所示，本实施例提供的现代应用的副本数据管理系统包括现代应用模块和副本数据管理模块。
现代应用模块包括业务单元、对象存储网关、生产对象存储和数据库，业务单元设置为生成大、小对象以及业务元数据，对象存储网关设置为在生产对象存储中写入和读取大对象、小对象，将小对象合并为大对象并生成对象元数据，数据库设置为存储业务元数据和对象元数据。对象元数据包括大对象名称、偏移量和长度，偏移量为小对象在大对象中的起始位置，长度包括小对象的长度。业务元数据是业务单元处理业务所需的数据业务元数据为业务单元所处理的数据，包括用户信息、用户权限、用户文件存储信息和对象名称。对象存储网关采用对象存储S3接口，访问生产对象存储以及灾备对象存储的地址改为对象存储网关的地址，使对生产对象存储以及灾备对象存储的操作全部由对象存储网关接管。
副本数据管理模块包括元数据处理单元和灾备对象存储，元数据处理单元设置为对元数据进行备份，生成并存储元数据副本，灾备对象存储设置为对生产对象存储进行备份，达到保护生产对象存储的目的。
元数据处理单元利用元数据副本及其快照实现现代应用模块的恢复以及副本利用。
所述的副本利用过程包括：
所述元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾备对象存储中的大对象，为第三方数据分析、数据挖掘、开发测试等系统提供数据，既保证了生产对象存储的安全，又充分利用了生产对象存储的数据。
业务单元还设置为向对象存储网关发送对象写请求和对象读请求；对象存储网关接收到对象写请求时，对象存储网关将大对象、小对象存储至生产对象 存储，当生产对象存储中的小对象数据量达到设定阈值时，对象存储网关将小对象合并为大对象；对象存储网关接收到对象读请求时，检测数据库中是否有对象元数据，若数据库中有对象元数据，则根据对象元数据或元数据副本访问大对象中指定位置以及长度的数据，若数据库中没有对象元数据，则直接读取大对象。
现代应用模块的恢复包括损坏恢复和历史恢复；
当生产对象存储损坏需要恢复时，需要进行损坏恢复，过程包括：将最新的元数据副本恢复至数据库，将灾备对象存储同步至生产对象存储；
当需要将生产对象存储恢复至历史版本时进行历史恢复，过程包括：元数据处理单元将元数据副本的快照恢复至数据库，元数据处理单元通过查询数据库查询对应的大对象，将该大对象恢复至生产对象存储，即可达到恢复到历史时间点的效果。
文档云中的许多用户有大量的图片、文档等文件，这些文件保存在生产对象存储中，为了提高生产对象存储保存数据的可靠性和安全性，也需要做数据保护，同时这些数据量庞大的数据有很大的价值，如做大数据分析、开发测试等；
实施方式如下：
部署对象存储网关，将生产对象存储和灾备对象存储的信息配置到对象存储网关中；将文档云中生产对象存储和灾备对象存储的地址修改为对象存储网关的地址；部署副本数据管理模块，将生产对象存储备份至灾备对象存储，数据库备份至元数据处理单元，完成数据的保护；将指定时间点的元数据快照克隆并挂载给数据库，并通过对象存储网关访问灾备对象存储，为第三方数据分析、数据挖掘、开发测试等系统提供数据。
本实施例中的大对象和小对象分别对应业务单元的大文件和小文件。
大文件为数据量大于一阈值的文件，小文件为数据量小于或等于该阈值的文件。
实施例2
与实施例1对应的一种现代应用的副本数据管理方法，包括：
业务单元生成大、小对象以及业务元数据，将业务元数据存储至数据库，并向对象存储网关发送对象写请求和对象读请求；
对象存储网关接收到对象写请求时将大、小对象存储至生产对象存储，当生产对象存储中的小对象数据量达到设定阈值时，对象存储网关将小对象合并为大对象，并生成对象元数据，将对象元数据存储至数据库，对象存储网关采用对象存储S3接口，灾备对象存储对生产对象存储进行备份，对象元数据包括大对象名称、偏移量和长度，业务元数据为业务单元所处理的数据，包括用户信息、用户权限、用户文件存储信息和对象名称；
对象存储网关接收到对象读请求时，检测数据库中是否有对象元数据，若数据库中有对象元数据，则根据对象元数据访问大对象中指定位置以及长度的数据，若数据库中没有对象元数据，则直接读取大对象；
元数据处理单元利用元数据副本及其快照实现数据库以及生产对象存储的恢复以及副本利用；
所述的副本利用过程为：
所述的元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾备对象存储中的大对象，为第三方数据分析、数据挖掘、开发测试等系统提供数据，既保证了生产对象存储的安全，又充分利用了生产对象存储的数据。
数据库以及生产对象存储的恢复包括损坏恢复和历史恢复；
损坏恢复过程包括：所述的元数据处理单元将最新的元数据副本恢复至数据库，将所述的生产对象存储与灾备对象存储同步；
历史恢复包括：所述的元数据处理单元将指定时间点的元数据副本的快照恢复至数据库，所述的灾备对象存储通过查询数据库查询对应的大对象，并将该大对象恢复至生产对象存储，即可达到恢复到历史时间点的效果。
实施例1和实施例2提出了一种现代应用的副本数据管理系统及方法，针对现代应用系统特性，通过副本数据管理技术，将现代应用系统的大、小对象、业务元数据以及对象元数据进行保护以及备份数据的再利用，充分发挥备份数据的价值，并通过实现一个对象存储网关，将小对象合并成大对象，使对象存储中数据能快速地迁移到副本数据管理系统中，在需要恢复的时候能快速地恢 复，大幅提升数据保护的性能，解决现代应用中的数据的保护和再利用问题。
本申请通过对象存储网关采用对象元数据访问的方式整合小对象，并对生产对象存储和生产对象数据库进行备份，对应获得灾备对象存储和元数据副本，对元数据副本依次进行快照和克隆，获得克隆元数据快照，恢复生产对象存储的速度快，利用克隆元数据快照挂载到灾备对象数据库，并通过灾备对象存储网关以只读方式访问灾备对象存储，能够充分利用灾备对象存储中存储的数据，同时又不影响生产对象存储的安全，安全性好。
本申请可对生产对象存储进行损坏恢复和历史恢复，进行损坏恢复时元数据处理单元将最新的元数据副本恢复至生产对象数据库，将灾备对象存储同步至生产对象存储，进行历史恢复时将指定时间点的元数据副本的快照恢复至生产对象数据库，灾备对象存储通过查询数据库查询对应的大对象并将该大对象恢复至生产对象存储，安全性好，操作简便。

Claims (10)

  1. 一种现代应用的副本数据管理系统，包括：
    现代应用模块，所述现代应用模块包括业务单元、对象存储网关、生产对象存储和数据库，所述业务单元与所述对象存储网关以及所述数据库连接，所述业务单元设置为生成大对象、小对象以及业务元数据，将所述大对象和所述小对象发送至所述对象存储网关以及将所述业务元数据存储至所述数据库中，所述对象存储网关与所述生产对象存储和所述数据库连接，所述生产对象存储网关设置为在生产对象存储中写入和读取大对象和小对象，将小对象合并为大对象并生成对象元数据以及将所述对象元数据存储至所述数据库，所述数据库设置为存储业务元数据和对象元数据；
    副本数据管理模块，所述副本数据管理模块包括元数据处理单元和灾备对象存储，所述元数据处理单元与所述数据库连接，所述元数据处理单元设置为对数据库进行备份，生成元数据副本，所述灾备对象存储与所述对象存储网关连接，所述对象存储网关还设置为将所述生产对象存储备份到所述灾备对象存储；
    所述元数据处理单元还设置为利用元数据副本及其快照实现现代应用模块的恢复以及副本利用；
    所述的副本利用过程为：
    所述元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾备对象存储中的大对象。
  2. 根据权利要求1所述的系统，其中，所述业务单元还设置为向对象存储网关发送对象写请求和对象读请求；
    所述对象存储网关是设置为：在接收到对象写请求的情况下，将接收到的大对象或小对象存储至生产对象存储，在生产对象存储中的小对象的数据量达到设定阈值的情况下，将小对象合并为大对象；在接收到对象读请求的情况下，检测数据库中是否有对象元数据，响应于数据库中有对象元数据的检测结果，根据对象元数据访问生产对象存储中的大对象中指定位置以及长度的数据，响应于生产对象数据库中没有对象元数据的检测结果，直接读取生产对象存储中 的大对象。
  3. 根据权利要求1所述的系统，其中，所述现代应用模块的恢复包括损坏恢复和历史恢复；
    所述损坏恢复过程包括：所述元数据处理单元将最新的元数据副本恢复至数据库，所述对象存储网关将所述生产对象存储与灾备对象存储同步；
    所述历史恢复过程包括：所述元数据处理单元将指定时间点的元数据副本的快照恢复至数据库，所述对象存储网关通过查询数据库查询与所述指定时间点对应的大对象，并将所述大对象恢复至生产对象存储。
  4. 根据权利要求1所述的系统，其中，所述业务元数据包括用户信息、用户权限、用户文件存储信息和对象名称，所述对象元数据包括大对象名称、偏移量和长度。
  5. 根据权利要求1所述的系统，其中，所述对象存储网关采用对象存储S3接口。
  6. 一种现代应用的副本数据管理方法，包括：
    业务单元生成大对象、小对象以及业务元数据，将所述大对象和所述小对象发送至所述对象存储网关以及将所述业务元数据存储至所述数据库中；对象存储网关在生产对象存储写入和读取大对象和小对象并将小对象合并为大对象，生成对象元数据，将对象元数据存储至数据库，元数据处理单元对数据库进行备份生成元数据副本，所述生产对象存储网关将所述生产对象存储备份到灾备对象存储；
    元数据处理单元利用元数据副本及其快照实现数据库和生产对象存储的恢复以及副本利用；
    所述副本利用过程包括：
    所述元数据处理单元对元数据副本依次进行快照和克隆，生成克隆元数据快照，将克隆元数据快照挂载到数据库并通过对象存储网关以只读方式读取灾备对象存储中的大对象。
  7. 根据权利要求6所述的方法，还包括，所述业务单元向对象存储网关发送对象写请求和对象读请求；
    所述对象存储网关在接收到对象写请求的情况下，所述对象存储网关将接收的大对象或小对象存储至生产对象存储，在生产对象存储中的小对象的数据量达到设定阈值的情况下，对象存储网关将小对象合并为大对象；
    所述对象存储网关在接收到对象读请求的情况下，所述对象存储网关检测数据库是否有对象元数据，响应于数据库中有对象元数据的检测结果，根据对象元数据访问生产对象存储中的大对象中指定位置以及长度的数据，响应于数据库中没有对象元数据的检测结果，直接读取生产对象存储中的大对象。
  8. 根据权利要求6所述的方法，其中，所述生产对象数据库和生产对象存储的恢复包括损坏恢复和历史恢复；
    所述损坏恢复过程包括：所述元数据处理单元将最新的元数据副本恢复至数据库，所述对象存储网关将所述生产对象存储与灾备对象存储同步；
    所述历史恢复包括：所述元数据处理单元将指定时间点的元数据副本的快照恢复至数据库，所述对象存储网关通过查询数据库查询与所述指定时间点对应的大对象，并将所述大对象恢复至生产对象存储。
  9. 根据权利要求6所述的方法，其中，所述业务元数据包括用户信息、用户权限、用户文件存储信息和对象名称，所述对象元数据包括大对象名称、偏移量和长度。
  10. 根据权利要求6所述的方法，其中，所述对象存储网关采用对象存储S3接口。
PCT/CN2020/124230 2020-04-14 2020-10-28 现代应用的副本数据管理系统及方法 WO2021208402A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/996,248 US20230214301A1 (en) 2020-04-14 2020-10-28 Copy Data Management System and Method for Modern Application

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010290216.1A CN111581016B (zh) 2020-04-14 2020-04-14 一种现代应用的副本数据管理系统及方法
CN202010290216.1 2020-04-14

Publications (1)

Publication Number Publication Date
WO2021208402A1 true WO2021208402A1 (zh) 2021-10-21

Family

ID=72122574

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/124230 WO2021208402A1 (zh) 2020-04-14 2020-10-28 现代应用的副本数据管理系统及方法

Country Status (3)

Country Link
US (1) US20230214301A1 (zh)
CN (1) CN111581016B (zh)
WO (1) WO2021208402A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111581016B (zh) * 2020-04-14 2021-05-18 上海爱数信息技术股份有限公司 一种现代应用的副本数据管理系统及方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2230601A1 (en) * 2009-03-19 2010-09-22 Symantec Corporation Method for restoring data from a monolithic backup
CN102457555A (zh) * 2010-10-28 2012-05-16 中兴通讯股份有限公司 一种分布式存储的安全系统及方法
CN104133882A (zh) * 2014-07-28 2014-11-05 四川大学 一种基于hdfs的小文件处理方法
CN106686148A (zh) * 2017-03-20 2017-05-17 郑州云海信息技术有限公司 一种用于提高对象存储系统中对象上传速度的方法和系统
CN109976941A (zh) * 2017-12-28 2019-07-05 华为软件技术有限公司 一种数据恢复方法和装置
CN111581016A (zh) * 2020-04-14 2020-08-25 上海爱数信息技术股份有限公司 一种现代应用的副本数据管理系统及方法

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8161318B2 (en) * 2005-02-07 2012-04-17 Mimosa Systems, Inc. Enterprise service availability through identity preservation
CN101854388B (zh) * 2010-05-17 2014-06-04 浪潮(北京)电子信息产业有限公司 一种集群存储中并行访问大量小文件的方法及系统
US8892845B2 (en) * 2010-12-22 2014-11-18 Cleversafe, Inc. Segmenting data for storage in a dispersed storage network
US9294564B2 (en) * 2011-06-30 2016-03-22 Amazon Technologies, Inc. Shadowing storage gateway
US9710475B1 (en) * 2012-07-16 2017-07-18 Tintri Inc. Synchronization of data
US9069955B2 (en) * 2013-04-30 2015-06-30 International Business Machines Corporation File system level data protection during potential security breach
WO2015081473A1 (zh) * 2013-12-02 2015-06-11 华为技术有限公司 异步复制方法、装置与系统
CN103838646B (zh) * 2014-02-13 2017-10-20 中国科学院国家天文台 一种用于地面应用大数据异地容灾备份的系统和方法
US9563518B2 (en) * 2014-04-02 2017-02-07 Commvault Systems, Inc. Information management by a media agent in the absence of communications with a storage manager
CN104978351A (zh) * 2014-04-09 2015-10-14 中国电信股份有限公司 海量小文件备份方法与云存储网关
US10565230B2 (en) * 2015-07-31 2020-02-18 Netapp, Inc. Technique for preserving efficiency for replication between clusters of a network
CN106790434B (zh) * 2016-12-05 2020-03-27 上海爱数信息技术股份有限公司 网络数据的管理方法、网络附属存储网关及存储服务系统
EP3714372A4 (en) * 2018-07-06 2021-02-24 Snowflake Inc. DATA REPLICATION AND DATA FAILURE BACKUP IN DATABASE SYSTEMS
CN109067733B (zh) * 2018-07-27 2021-01-05 成都华为技术有限公司 发送数据的方法和装置，以及接收数据的方法和装置
US11341234B1 (en) * 2019-06-05 2022-05-24 EMC IP Holding Company LLC System for securely recovering backup and data protection infrastructure
US11625306B2 (en) * 2020-11-04 2023-04-11 Netapp, Inc. Data connector component for implementing data requests
US11567835B2 (en) * 2021-04-25 2023-01-31 Kyndryl, Inc. Data protection and recovery
US20230083104A1 (en) * 2021-09-10 2023-03-16 Vmware, Inc. Efficiently Deleting Snapshots in a Log-Structured File System (LFS)-Based Storage System
US11663083B2 (en) * 2021-10-29 2023-05-30 EMC IP Holding Company LLC Cyber-related data recovery

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2230601A1 (en) * 2009-03-19 2010-09-22 Symantec Corporation Method for restoring data from a monolithic backup
CN102457555A (zh) * 2010-10-28 2012-05-16 中兴通讯股份有限公司 一种分布式存储的安全系统及方法
CN104133882A (zh) * 2014-07-28 2014-11-05 四川大学 一种基于hdfs的小文件处理方法
CN106686148A (zh) * 2017-03-20 2017-05-17 郑州云海信息技术有限公司 一种用于提高对象存储系统中对象上传速度的方法和系统
CN109976941A (zh) * 2017-12-28 2019-07-05 华为软件技术有限公司 一种数据恢复方法和装置
CN111581016A (zh) * 2020-04-14 2020-08-25 上海爱数信息技术股份有限公司 一种现代应用的副本数据管理系统及方法

Also Published As

Publication number Publication date
US20230214301A1 (en) 2023-07-06
CN111581016B (zh) 2021-05-18
CN111581016A (zh) 2020-08-25

Similar Documents

Publication Publication Date Title
US20210089502A1 (en) Application-aware and remote single instance data management
US20220318190A1 (en) Image level copy or restore, such as image level restore without knowledge of data object metadata
US11429305B2 (en) Performing backup operations using replicas
US8219524B2 (en) Application-aware and remote single instance data management
US20160048427A1 (en) Virtual subdirectory management
US20120197844A1 (en) Block level data replication
EP3223158B1 (en) Data recovery method and storage system
US12001452B2 (en) Search and analytics for storage systems
US7483926B2 (en) Production server to data protection server mapping
WO2021208400A1 (zh) 现代应用的备份恢复系统及方法
US20070294310A1 (en) Method and apparatus for storing and recovering fixed content
US10484179B1 (en) Data consistency in an encrypted replication environment
US10146637B1 (en) Intelligent snapshot rollbacks
CN109828951B (zh) 一种结合Kubernetes和Ceph存储快照的方法及系统
US10877855B2 (en) Techniques for data backup and restoration
CN112800019A (zh) 基于Hadoop分布式文件系统的数据备份方法及系统
WO2021208402A1 (zh) 现代应用的副本数据管理系统及方法
WO2021208401A1 (zh) 现代应用的持续数据保护系统及方法
CN112346907B (zh) 一种基于异构对象存储的数据备份恢复方法及系统
CN103810056A (zh) 一种在Linux平台上获得一致性备份的方法
US11645333B1 (en) Garbage collection integrated with physical file verification
CN113672436A (zh) 一种容灾备份方法、装置、设备及存储介质
US11836051B2 (en) Method and system for protecting data during data migration
US10078641B1 (en) Optimized lock detection in a change block tracker
CN111382137A (zh) 一种Hadoop集群文件备份系统及方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20931434

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20931434

Country of ref document: EP

Kind code of ref document: A1