CN112380050A

CN112380050A - Method for using snapshot in system backup

Info

Publication number: CN112380050A
Application number: CN202011137643.2A
Authority: CN
Inventors: 邬玉良
Original assignee: Zhongke Hot Standby Beijing Cloud Computing Technology Co ltd
Current assignee: Zhongke Hot Standby Beijing Cloud Computing Technology Co ltd
Priority date: 2020-10-22
Filing date: 2020-10-22
Publication date: 2021-02-19

Abstract

The invention relates to an application method of a snapshot in system backup, which aims to ensure the continuity of services in computer data filing, storage, backup, remote disaster recovery and disaster recovery by creating the snapshot, saving the snapshot, recovering the snapshot and other technologies in the backup process, thereby achieving the shortest recovery time and the smallest data loss (the smallest RTO and RPO).

Description

Method for using snapshot in system backup

Technical Field

The invention relates to an application method of a snapshot in system backup, belonging to the field of computer data processing.

Background

In recent years, the demand of using computers for data storage is increasing, and with the increase of the demand of storage applications, user data is easily damaged and lost, and a snapshot is one of effective methods for preventing data loss of a storage device, and in the past decade, the snapshot has become a standard configuration function of a disk array, a volume manager, a file system, and even a PC I RAID (redundant array of independent disks) controller. More and more devices are beginning to support this functionality. More and more storage devices support the snapshot function, and the advantages of respective snapshot technologies are promoted in the data of the products, some are more snapshots, and some occupy small space.

The definition of snapshots by SNIA (storage network industry association) is: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the point in time at which the copy begins). The snapshot may be a copy of the data it represents or may be a replica of the data. Whereas, in particular technical details, a snapshot is a reference mark or pointer to data stored in a storage device. We can understand that a snapshot is somewhat like an inventory table, but it is treated by the computer as a full backup of data.

There are three basic forms of snapshots: file system based, subsystem based and volume manager/virtualization based, and the three forms are very different. A utility tool capable of automatically generating these snapshots has appeared in the market, such as a representative storage device with NetApp is implemented based on a file system, and high, medium and low end devices can implement snapshot applications using a common operating system.

The snapshot technology functions as follows: the method mainly can carry out online data recovery, and can carry out timely data recovery when application failure or file damage occurs to the storage equipment, so as to recover the data to the state of a snapshot generating time point. The snapshot has another function of providing another data access channel for the storage user, so that when the original data is subjected to online application processing, the user can access the snapshot data and can also utilize the snapshot to perform work such as testing. Therefore, all storage systems, whether high, medium, and low, are applied to online systems, and snapshot becomes an indispensable function.

There are two broad classes of storage snapshots, one called write-and-copy snapshots and the other called split mirror snapshots.

That is, the write-and-copy snapshot may generate a snapshot of changes to stored data each time new data is input or existing data is updated. This allows data to be quickly recovered in the event of a hard disk write error, file corruption, or program failure. However, if full archiving or restoration of all data on the network or storage medium is desired, all previous snapshots must be available for use. I.e., a write-and-copy snapshot is a "photo" that characterizes the appearance of data. This approach is also commonly referred to as "metadata" copying, i.e., all the data is not actually copied to another location, but only a pointer indicating where the data is actually located is copied. With this technique, when a snapshot is already available, if someone attempts to overwrite the data on the original LUN, the snapshot software will first copy the original data block to a new location (the pool of storage resources dedicated to the copy operation) and then perform the write operation. The snapshot software later maps the pointer to a new location when you reference the original data, or to an old location when you reference the snapshot.

The split mirror snapshot references all data on the mirror hard disk group. Each time the application runs, a snapshot of the entire volume is generated, not just the new or updated data. This enables off-line access to the data and simplifies the process of recovering, copying or archiving all the data on one hard disk. However, this is a slow process and each snapshot needs to take up more storage space, also called copy-as-is, since it is a physical copy of data on a certain LUN or file system, some administrators are called clones, images, etc. The process of copying as is may be done by the host or in hardware at the storage level.

The snapshot technology uses the method: when snapshots are used in particular, the storage administrator may have three forms, namely a cold snapshot copy, a warm snapshot copy, and a hot snapshot copy.

Copying a cold snapshot: making a cold snapshot copy is the safest way to ensure that the system can be fully restored. Cold copies are typically required before and after any major configuration changes or maintenance procedures are performed to ensure complete recovery (rolback). Cold copy may also be combined with cloning techniques to replicate an entire server system for various purposes, such as extending, making a replica of the production system for testing/development, and migrating to two-tier storage.

Copying the warm snapshot: the warm snapshot copy utilizes the suspend function of the server. When the suspend action is performed, the program counter is stopped, all active memory is saved in a temporary file (.vmss file) in the file system where the boot hard disk resides, and the server application is suspended. At this point in time, a snapshot copy of the entire server (including the memory content files and all LUNs and associated active file systems) is replicated. In this copy, the machine and all data will be frozen at the point of processing when the suspend operation is completed.

When the snapshot operation is complete, the server may be restarted, resuming execution at the point where the suspend action began. The application and server processes will resume running from the same point in time. It appears superficially as if a pause key was pressed during the snapshot activity. To the network client of the server, it appears as if the network service was temporarily interrupted. For a moderately loaded server, this time is typically 30 to 120 seconds.

Copying hot snapshots: in this state all write operations that occur are immediately applied to a virtual hard disk to maintain a high degree of consistency of the file system. The server provides a tool to put the persistent virtual hard disk in a hot backup mode to replicate snapshot copies on the hard disk subsystem layer by adding REDO log files.

Once the REDO log is activated, it is safe to replicate the snapshot containing the LUNs of the server file system. After the snapshot operation is complete, another command may be issued that submits REDO log processing to the underlying virtual hard disk file. When the commit activity is complete, all log entries will be applied and the REDO file will be deleted. During the execution of this operation, a slight decrease in processing speed occurs, but all operations will continue to be executed. However, in most cases, the snapshot process is almost instantaneous and the time between creation and commit of the REDO is very short.

The hot snapshot operation process is seemingly substantially imperceptible of a server speed drop. In the worst case, it appears to be a general server speed drop that may be caused by a CPU that is congested or overloaded with the network. At best, no perceptible effects occur.

Snapshot technique-distinction between snapshot and mirroring, replication

Mirroring, snapshotting and copying are three different functions

Mirroring is the copying of data by creating two I/Os from one I/O. The disk image is created on the host system by the OS or volume management software. Disk mirroring is a local option that relies on platform and local connectivity characteristics. Mirroring can be used for DAS and SAN and most NAS support it. Storage-forwarding mirror disk subsystems (e.g., EMC SRDF, IBM PPRC, Hitachi TrueCopy) are mainly used for SAN products.

Replication is the transmission of data objects (files, tables, etc.) over a network. The transfer is from system to system rather than between storage devices or subsystems. Replication is also generally platform specific, so the way in which Windows 2000 replication products operate is very different from that of the Unix platform.

Data snapshot technology in snapshot technology-disaster recovery technology

The remote mirroring technology is often combined with the snapshot technology to implement remote backup, that is, data is backed up to a remote storage system by mirroring, and then information in the remote storage system is backed up to a remote tape library or an optical disk library by using the snapshot technology.

The snapshot technology is divided into two types, one type is pointer type, and is characterized in that the data of a disk subsystem to be backed up is quickly scanned by software to establish a snapshot logical unit number LU of the data to be backed up

Whether production, test, disaster recovery or data warehouse application, the system component can provide generation, management and maintenance of business data image copies, so that customers can better and more fully utilize business data in different modes, and greater value-added benefits are obtained. Copy management system components designed based on data copying, protection and information sharing provide very powerful functions: the method can realize data replication in one data center or among different data centers, can realize information sharing among different departments, quickly and effectively adapts to flexible expansion of services, and obtains leading advantages in competition. The method can enable the user to extract the current online service data in real time under the condition that the normal service is not influenced. The 'backup window' is close to zero, which can greatly increase the continuity of system service and provide guarantee for realizing the real 7 x 24 operation of the system.

Disclosure of Invention

The invention relates to a backup method aiming at a snapshot technology utilized in a backup process, which aims to solve the problems of low utilization rate of computer storage space and reduction of misjudgment probability of data deletion in computer data archiving, storage, backup, remote disaster recovery and disaster recovery.

The invention provides a user-based snapshot technology idea, namely different data backup mechanisms are adopted according to the data importance level of enterprise users. A snapshot system capable of online hot backup is designed and implemented. The design of the snapshot system uses the principle of incremental backup, the snapshot version is dependent on the previous snapshot version, and the data which is not updated between the snapshot points is shared between the snapshot versions. In the design, each snapshot version marks data update after a snapshot point arrives in an incremental bitmap mode, the snapshot data update is realized by data redirection, and compared with the method adopting a copy-before-write technology, the data redirection method does not increase IO load of a system. The snapshot data mapping maintenance adopts a hash table mode, and the redirected user request can be searched in constant time. Based on user management, the snapshot system distributes snapshot volume according to user priority, and determines snapshot generation time interval and volume of snapshot volume according to user priority. And finally, respectively carrying out function test and performance test on the realized snapshot system. The snapshot system has realized that the user establishes the snapshot voluntarily, the snapshot resumes and the snapshot deletes the function, the test result of the snapshot function shows, the snapshot establishing time can reach dozens of microseconds.

The invention provides a backup method of a snapshot mode, which comprises the following steps:

first, a basic module of a client is installed in the application service period, as if a data filter is added to a server

Secondly, when the user writes data into the hard disk, the data is written in a byte mode, before the data is written into the hard disk, the filter knows the written value of the data, and the data is transmitted into the storage pool of the server through the network under the condition of write IO which does not affect the service period. The method for writing the database into the hard disk is characterized in that the database is written into the hard disk in a byte mode by taking complete transactions as units according to own rules, at this time, the modes of writing the database into the local disk and writing the database into the storage pool are completely consistent, the writing mode of the database simulates the writing mode of the database in the local hard disk, and the strict consistency of the database data is ensured by the simulated writing mode of the database. Because the data written into the CDP storage pool is not required to scan a database storage hard disk of the application server, but is directly written into the CDP storage pool according to the same mode, the data synchronization is rapid, and the consistency and the integrity of the service data are ensured.

Drawings

FIG. 1 is a diagram of a backup method based on snapshot technology

Detailed Description

1: establishing data mount points

1) All nodes install sshfs, can be installed through yum sources, and depend on epel;

2) after sshfs is installed, planning a shared directory mount point, determining a target mount point, and if the read-write permission needs to be distributed to the shared directory of each node, directly accessing a chmod 777-shared directory path;

3) mounting the shared directory to the target host path by:

#sshfs root@10.200.XX.XXX:/data1/esdata/data1/backups-o allow_other

the path [ user name @ IP address ] in the instruction is an actual mounting node path and a user, and the latter path is a local node mounting path;

4) mounting the same path on each node of the ES to the same remote address, and after the same path is mounted to the same target host, confirming that the backup directories of all the nodes are mounted to the path of the same target host;

2: establishing snapshot backup

1) Establishing snapshot through API of ES:

the parameters here specify the index that needs the snapshot (wildcard supported), the policy (ignore) that encounters an index that is not available, and the option to not backup the common information, respectively.

2) Verifying snapshots

#curl

http://10.200.XXX.XXX:9200/_snapshot/backup_38/snapshot_20160628| python-mjson.tool

The snapshot information just established will normally be returned.

3: recovering snapshot data

1) Copying backup files

The contents in the mount directory of the file server all belong to the contents needing to be backed up.

2) Establishing a shared file system

Establishing a shared system on the host that copied the backup snapshot information in the same manner, the steps being the same as before;

3) establishing snapshot storage repository

Warehouse reposititory is built in the same way;

4) data recovery

Data is recovered with the following APIs:

curl-XPOST

http://192.168.0.1:9200/snapshot/backup_38/snapshot_20160628/restore

the recovery can also be customized by adding the following parameters:

{

"indices":"index1",

"rename_pattern":"index_(.+)",

"rename_replacement":"restored_index_$1"

}

5) viewing data recovery progress

curl-XGET

http://192.168.0.1:9200/snapshot/backup_38/snapshot_20160628/status。

Claims

1. A method for using snapshots in a system backup, characterized by:

creating a snapshot in the system backup first, and saving the snapshot based on the initial state;

after the snapshot foundation of the first step is available, establishing a mirror image relationship between the source end and the destination end, completing the initialization synchronization work from the source end to the destination end, and then storing the snapshot in a local storage;

judging whether the data organization structure to be processed is consistent with the existing data or not;

if the data are consistent, acquiring the data of the structure type, loading the data from the hard disk to the memory, and transmitting the data to be processed; if the data are inconsistent, acquiring the data of the data structure, synchronizing the data to the snapshot space, and transmitting the data to be processed;

correspondingly setting the data to be processed according to factors such as requirements and storage performance, and carrying out snapshot operation at certain time intervals;

and extracting the initial state of the snapshot in the storage, and comparing the initial state with the destination, wherein the operations of the 4 th step and the 5 th step can be synchronously performed.

2. The method of claim 1, wherein the next recovery operation is determined based on an initial state of the source data snapshot.

3. The method of claim 1, wherein the purpose of comparing the initial snapshots is to compare data of a source end and a destination end, and ensure consistency of the data by ensuring that the two ends are the same during data storage.

4. The method of claim 2, wherein a recoverable base is used to ensure the inconsistency of subsequent data according to the state of the initial snapshot of the source end.

5. A method as claimed in claim 1, characterized in that it is implemented in the backup process in several phases separated by a certain time period by the snapshot implementation of the source data and the synchronous transmission of the snapshot state to the destination.

6. The method as recited in claim 1, wherein the snapshot state comparisons returned upon completion are compared to ensure data consistency.