CN117215830A

CN117215830A - Data backup method, device, computer equipment and computer readable storage medium

Info

Publication number: CN117215830A
Application number: CN202311134758.XA
Authority: CN
Inventors: 陈德虎
Original assignee: Pingan Payment Technology Service Co Ltd
Current assignee: Pingan Payment Technology Service Co Ltd
Priority date: 2023-09-04
Filing date: 2023-09-04
Publication date: 2023-12-12

Abstract

The application discloses a data backup method, a device, computer equipment and a computer readable storage medium, which relate to the computer technology, and can identify whether over-backup is initiated on the same backup by inquiring the number of the backup libraries of a cluster and acquiring backup information through metadata, save disk space, reduce the maintenance and adjustment difficulty of backup tasks through metadata even if Redis is changed by main-backup switching, machine half-replacement and the like, improve the backup accuracy and improve the backup efficiency. The method comprises the following steps: acquiring a backup task to be executed, and determining a main database to be backed up; inquiring metadata information of the database cluster, and determining a plurality of backup databases with association relations with the main database; determining a target backup database, and executing data backup operation on the main database based on the target backup database to obtain a backup result; and generating a database backup record comprising the backup result, and storing the database backup record in a backup log table.

Description

Data backup method, device, computer equipment and computer readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data backup method, apparatus, computer device, and computer readable storage medium.

Background

With the rapid development of computer networks, the importance of information security is becoming more and more obvious, and data backup is an important part of information security and is an important means for protecting data because data is possibly lost or even cannot be estimated or compensated in the processes of data transmission, data storage and data exchange. Redis (Remote Dictionary Server, remote dictionary service) is a non-relational database that runs on memory and can persist data, save data in key-value pair format, and save data in log-type files. The Redis database supports master-slave synchronization, can store data which is frequently used, reduces the times of accessing the database, and improves the operation efficiency.

In the related art, redis is often a master and multiple standby, such as a master and three standby, and the backup task is controlled by a timing task Crontab (set periodically executed instruction) of the operating system. In order to prevent the roles from changing after the primary and backup are switched, a backup task needs to be deployed at each node, and whether to initiate backup is determined by judging the primary and backup roles, wherein the backup is not initiated by the default primary library, and the backup library allows the backup. However, the applicant realizes that if each node has a backup task, backup is performed on three backup libraries, but the data of the three backup libraries are identical, so that repeated backup occupies three disk spaces, which causes resource waste, the backup tasks are deployed through Crontab, once data switching occurs, the maintenance and adjustment difficulty of the backup tasks in the Redis cluster is high, the number of the backup libraries in the Redis cluster cannot be determined, whether the backup is initiated on the same backup library cannot be identified, and the backup accuracy is low and the backup efficiency is low.

Disclosure of Invention

In view of this, the present application provides a data backup method, apparatus, computer device and computer readable storage medium, and mainly aims to solve the problems of resource waste caused by repeated backup of Redis clusters, and low backup accuracy and low backup efficiency caused by high difficulty in maintaining and adjusting backup tasks.

According to a first aspect of the present application, there is provided a data backup method, the method comprising:

acquiring a backup task to be executed, and determining a target main database to be backed up indicated by the backup task to be executed;

inquiring metadata information of a database cluster where the target main database is located, and determining a plurality of backup databases in association with the target main database in the database cluster according to a database topological structure included in the metadata information, wherein the database topological structure indicates association between a plurality of main databases and a plurality of backup databases included in the database cluster;

determining a target backup database in the plurality of backup databases, and executing data backup operation on the target main database based on the target backup database to obtain a backup result;

And generating a database backup record comprising the backup result, and storing the database backup record in a backup log table.

Optionally, before the obtaining the backup task to be executed and determining the target primary database to be backed up indicated by the backup task to be executed, the method further includes:

determining a database cluster to be backed up, and acquiring metadata information of the database cluster;

determining a plurality of databases included in the database cluster, inquiring attribute information corresponding to each database in the plurality of databases in the metadata information, and extracting at least one main database from the plurality of databases according to the attribute information;

reading a main database identifier of each main database in the at least one main database in the metadata information, acquiring a preset backup time interval, and generating a backup task to be executed carrying the main database identifier of the main database for each main database at intervals of the preset backup time interval;

when the number of the backup tasks to be executed is multiple, the backup tasks to be executed are formed into a backup task queue to be executed, and an asynchronous execution thread is called, so that the backup tasks to be executed are executed asynchronously.

Optionally, determining a target backup database from the plurality of backup databases, and performing a data backup operation on the target primary database based on the target backup database to obtain a backup result, including:

selecting a backup database from the plurality of backup databases as the target backup database, wherein the target backup database is currently in an idle state or the current residual data storage space is larger than or equal to the data quantity to be backed up of the target main database;

reading main database address information of the target main database in the metadata information, and reading backup database address information of the target backup database in the metadata information;

controlling the target backup database to establish connection with the target main database according to the main database address information and the backup database address information;

acquiring a preset backup script, calling the preset backup script to extract data to be backed up from the target main database, and storing the data to be backed up into a distributed storage space of the target backup database based on the preset backup script;

And when the completion of the execution of the data backup operation is detected, calling the preset backup script to acquire a backup result of the data backup operation, wherein the backup result indicates the success or failure of the execution of the data backup operation.

Optionally, the determining a target backup database from the plurality of backup databases, performing a data backup operation on the target primary database based on the target backup database, and after obtaining a backup result, the method further includes:

when the backup result indicates that the execution of the current data backup operation fails, acquiring a main database identifier of the target main database and a backup database identifier of the target backup database, and generating a backup failure reminder carrying the main database identifier and the backup database identifier;

sending the backup failure prompt to terminal equipment where a maintainer is located, so that the maintainer repairs the target main database and the target backup database according to the backup failure prompt;

when a data backup instruction initiated after repair of the maintenance personnel is completed is received, acquiring the main database identifier and the backup database identifier indicated by the data backup instruction, determining a target main database indicated by the main database identifier and a target backup database indicated by the backup database identifier, and re-executing data backup operation on the target main database based on the target backup database to obtain a backup result.

Optionally, the generating a database backup record including the backup result, and storing the database backup record in a backup log table includes:

acquiring the backup completion time of the data backup operation, and acquiring the backup database identifier of the target backup database and the main database identifier of the target main database from the metadata information;

generating a database backup record comprising the backup completion time, the backup database identifier, the main database identifier and the backup result;

and acquiring a backup log table of the database cluster, and storing the database backup record in the backup log table.

Optionally, the method further comprises:

acquiring a plurality of database backup records included in the backup log table;

generating record detailed information according to the content included in the database backup records for each database backup record in the plurality of database backup records, extracting a main database identifier from the database backup records, and associating the record detailed information with the main database identifier;

acquiring a plurality of record detailed information of the database backup records, wherein each record detailed information in the plurality of record detailed information is associated with a main database identifier;

Generating a detailed information page, adding the plurality of recorded detailed information to the detailed information page, and pushing the detailed information page to a terminal when detecting that the terminal requests to display the detailed information page.

Optionally, the method further comprises:

taking the time point of the acquired backup result as a backup completion time point, and acquiring a preset transfer time interval;

when the time interval between the current time and the backup completion time point reaches the preset dump time interval, acquiring backup data which is completed in the distributed storage space of the target backup database;

and the backup data are transferred to a database storage medium of the target backup database.

According to a second aspect of the present application, there is provided a data backup apparatus comprising:

the acquisition module is used for acquiring a backup task to be executed and determining a target main database to be backed up indicated by the backup task to be executed;

the query module is used for querying metadata information of a database cluster where the target main database is located, and determining a plurality of backup databases in association with the target main database in the database cluster according to a database topological structure included in the metadata information, wherein the database topological structure indicates association between a plurality of main databases and a plurality of backup databases included in the database cluster;

The backup module is used for determining a target backup database from the plurality of backup databases, and executing data backup operation on the target main database based on the target backup database to obtain a backup result;

and the storage module is used for generating a database backup record comprising the backup result and storing the database backup record in a backup log table.

Optionally, the apparatus further comprises:

the generation module is used for determining a database cluster to be backed up and acquiring metadata information of the database cluster; determining a plurality of databases included in the database cluster, inquiring attribute information corresponding to each database in the plurality of databases in the metadata information, and extracting at least one main database from the plurality of databases according to the attribute information; reading a main database identifier of each main database in the at least one main database in the metadata information, acquiring a preset backup time interval, and generating a backup task to be executed carrying the main database identifier of the main database for each main database at intervals of the preset backup time interval; when the number of the backup tasks to be executed is multiple, the backup tasks to be executed are formed into a backup task queue to be executed, and an asynchronous execution thread is called, so that the backup tasks to be executed are executed asynchronously.

Optionally, the backup module is configured to select a backup database from the plurality of backup databases as the target backup database, where the target backup database is a backup database currently in an idle state or a current remaining data storage space is greater than or equal to a data amount to be backed up of the target main database; reading main database address information of the target main database in the metadata information, and reading backup database address information of the target backup database in the metadata information; controlling the target backup database to establish connection with the target main database according to the main database address information and the backup database address information; acquiring a preset backup script, calling the preset backup script to extract data to be backed up from the target main database, and storing the data to be backed up into a distributed storage space of the target backup database based on the preset backup script; and when the completion of the execution of the data backup operation is detected, calling the preset backup script to acquire a backup result of the data backup operation, wherein the backup result indicates the success or failure of the execution of the data backup operation.

Optionally, the apparatus further comprises:

the restoration module is used for acquiring a main database identifier of the target main database and a backup database identifier of the target backup database when the backup result indicates that the execution of the current data backup operation fails, and generating a backup failure reminder carrying the main database identifier and the backup database identifier; sending the backup failure prompt to terminal equipment where a maintainer is located, so that the maintainer repairs the target main database and the target backup database according to the backup failure prompt; when a data backup instruction initiated after repair of the maintenance personnel is completed is received, acquiring the main database identifier and the backup database identifier indicated by the data backup instruction, determining a target main database indicated by the main database identifier and a target backup database indicated by the backup database identifier, and re-executing data backup operation on the target main database based on the target backup database to obtain a backup result.

Optionally, the storage module is configured to obtain a backup completion time when the data backup operation is performed, and obtain a backup database identifier of the target backup database and a master database identifier of the target master database in the metadata information; generating a database backup record comprising the backup completion time, the backup database identifier, the main database identifier and the backup result; and acquiring a backup log table of the database cluster, and storing the database backup record in the backup log table.

Optionally, the apparatus further comprises:

the display module is used for acquiring a plurality of database backup records included in the backup log table; generating record detailed information according to the content included in the database backup records for each database backup record in the plurality of database backup records, extracting a main database identifier from the database backup records, and associating the record detailed information with the main database identifier; acquiring a plurality of record detailed information of the database backup records, wherein each record detailed information in the plurality of record detailed information is associated with a main database identifier; generating a detailed information page, adding the plurality of recorded detailed information to the detailed information page, and pushing the detailed information page to a terminal when detecting that the terminal requests to display the detailed information page.

The transfer module is used for taking the time point of the acquired backup result as the backup completion time point and acquiring a preset transfer time interval; when the time interval between the current time and the backup completion time point reaches the preset dump time interval, acquiring backup data which is completed in the distributed storage space of the target backup database; and the backup data are transferred to a database storage medium of the target backup database.

According to a third aspect of the present application there is provided a computer device comprising a memory storing a computer program and a processor implementing the steps of the method of any of the first aspects described above when the computer program is executed by the processor.

According to a fourth aspect of the present application there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method of any of the first aspects described above.

By means of the technical scheme, the data backup method, the device, the computer equipment and the computer readable storage medium provided by the application are used for acquiring the backup task to be executed, determining the target main database to be backed up indicated by the backup task to be executed, inquiring the metadata information of the database cluster where the target main database is located, determining a plurality of backup databases in association with the target main database in the database cluster according to the database topology structure included in the metadata information, wherein the database topology structure indicates the association between the plurality of main databases and the plurality of backup databases included in the database cluster, determining the target backup database in the plurality of backup databases, executing data backup operation on the target main database based on the target backup database, obtaining the backup result, generating a database backup record comprising the backup result, storing the database backup record in a backup log table, inquiring the number of backup databases in the cluster through the metadata information, and acquiring backup information, and only carrying out backup on one backup database, thereby saving disk space, simultaneously identifying whether backup is initiated on the same backup database, and changing the backup information, even though the backup is changed, the backup information is changed, the accuracy of the machine can be improved, and the backup efficiency can be maintained.

The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present application more readily apparent.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:

FIG. 1A is a schematic flow chart of a data backup method according to an embodiment of the present application;

FIG. 1B is a schematic flow chart of a conventional data backup method;

FIG. 2A is a schematic flow chart of a data backup method according to an embodiment of the present application;

FIG. 2B is a schematic diagram illustrating a data backup architecture according to an embodiment of the present application;

FIG. 3A is a schematic diagram illustrating a data backup structure according to an embodiment of the present application;

FIG. 3B is a schematic diagram illustrating a data backup structure according to an embodiment of the present application;

Fig. 4 shows a schematic device structure of a computer device according to an embodiment of the present application.

Detailed Description

Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

The embodiment of the application provides a data backup method, as shown in fig. 1A, comprising the following steps:

101. and acquiring a backup task to be executed, and determining a target main database to be backed up indicated by the backup task to be executed.

The Redis cluster is used as an internal memory database, the high availability of the database cluster can be ensured through a main-standby mode, such as a main-standby mode and a standby mode, the main database performs read-write operation or performs write-only operation, and the standby database performs read-only operation. High availability refers to improving the availability of systems and applications by minimizing downtime caused by routine maintenance operations and sudden system crashes. The data of the main database is notified to the backup database after being changed, so that the backup database updates the data, and if one server is paralyzed, the backup node of the main node on the server is on the other server, and the database is kept available. However, the existing backup task is controlled by the timing task Crontab of the operating system, and in order to prevent the influence of role change on data after the primary and backup are switched, the backup task is deployed at each database node. As shown in fig. 1B, one existing data backup method is as follows:

In the active-standby mode of the Redis cluster, the backup task is controlled by the Crontab, for example, M1, M2 and M3 form a Redis cluster, and each primary database has three backup databases, for example, the backup database of the primary database M1 is S1_1, S1_2 and S1_3, the backup database of the primary database M2 is S2_1, S2_2 and S2_3, and the backup database of the primary database M3 is S3_1, S3_2 and S3_3. Crontab is a task performed at specified time intervals for maintenance and management of an automated system, such as backing up data, checking disk space usage, etc. Therefore, in order to prevent the influence of the role switching of the primary and the backup on the data, the backup task is deployed at each database node, and when the backup is performed, whether the backup is initiated or not is determined by judging the primary and the backup roles of the databases, wherein the backup is not initiated by the default primary database, such as M1, M2 and M3, and the backup database is allowed to be backed up. If each node has a backup task, three backup databases are backed up, for example, s1_1, s1_2, s1_3 all perform a backup operation on M1, s2_1, s2_2, s2_3 all perform a backup operation on M2, and s3_1, s3_2, s3_3 all perform a backup operation on M3.

However, the applicant realizes that when the method described in fig. 1B is adopted to perform backup, if each database node has a backup task, backup is performed on all three backup databases, and the data of the three backup databases are identical, so that the backup is repeatedly performed three times, and three disk spaces are occupied, thereby causing resource waste. Therefore, the technical scheme of the application can be applied to a data service platform, the data service platform can be operated based on an independent server, and can also be operated based on a server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (Content Delivery Network, CDN), big data, an artificial intelligent platform and other basic cloud computing, so that Redis clusters process data backup requests at a higher speed, and accurate backup is realized.

In the embodiment of the application, the data service platform initiates a backup task to be executed for backing up the data of the main database, and in order to determine which main database backup operation is executed after initiating the backup task to be executed, the backup task to be executed carries the main database identifier. Therefore, through the backup task to be executed, the data service platform can quickly and accurately determine the target main database to be backed up, so that the target backup database is determined according to the target main database later, the backup operation is executed on the target main database based on the target backup database, and the backup efficiency is improved.

102. Inquiring metadata information of a database cluster where a target main database is located, and determining a plurality of backup databases in association relation with the target main database in the database cluster according to a database topological structure included in the metadata information.

Wherein the metadata information is data describing data, or structural data for providing related information of a certain resource. The metadata information includes a database topology structure and attribute information of each database, wherein whether the database is a primary database or a backup database can be determined by querying the attribute information of each database, and association relations between a plurality of primary databases and a plurality of backup databases included in the database cluster can be determined by the database topology structure. Furthermore, information, such as address information, of each database in the database cluster can be obtained through metadata information. Therefore, the data service platform can acquire a plurality of backup databases associated with the main database in the database cluster through inquiring the database topological structure of the metadata information so as to further select the target backup database later, and the execution speed of the backup task to be executed can be improved.

In the embodiment of the application, the data service platform queries the database cluster where the target main database is located and determines the database cluster where the database belongs. Then, the data service platform inquires a database topological structure included in metadata information of the database cluster, and determines a plurality of backup databases in association relation with a target main database in the database cluster according to the database topological structure. The database topology indicates the association relationship between a plurality of main databases and a plurality of backup databases included in the database cluster, so that the plurality of backup databases associated with the main database can be rapidly and accurately acquired for the target main database. And if the target backup database fails, the data service platform can acquire a plurality of backup databases according to the database topology structure of the metadata information, so that one backup database is selected from the plurality of backup databases to adjust the target backup database, and normal operation of backup operation is ensured. Furthermore, the data service platform can accurately determine the number of backup databases with association relation with the main database through the database topological structure of the metadata information, can recognize whether the backup is initiated on the same backup database, and is used for determining a target backup database in a database cluster for the subsequent data service platform and executing the backup operation of the main database through the target backup database as a basis, so that the maintenance and adjustment difficulty of the data service platform to the backup task is reduced.

103. And determining a target backup database in the plurality of backup databases, and executing data backup operation on the target main database based on the target backup database to obtain a backup result.

In order to avoid the occupation of a large amount of storage space caused by repeated backup operation, the application uses the metadata information of the database cluster, selects one of the backup databases in the association relation with the main database as the target backup database to execute the backup operation, and can orderly carry out the whole backup process while improving the backup speed of the database cluster. And even if the primary and backup switch occurs in the database cluster, the data service platform can accurately determine that the target backup database executes the backup operation by using the metadata information, so as to realize accurate backup, and therefore, simple and efficient management of a large amount of networked data can be realized through the metadata information. In the embodiment of the application, the data service platform determines the target backup database in a plurality of backup databases which are in association with the target main database, so that repeated backup is avoided to occupy a large amount of storage space, and the backup rate can be improved. And then, based on the target backup database, performing data backup operation on the target main database to obtain a backup result, so that the data of the target main database can be timely and effectively saved, and the stable operation of the data service platform can be ensured while the data safety is ensured.

104. A database backup record is generated that includes the backup result and is stored in a backup log table.

The database backup record is used for recording information such as a main database, a backup database and a backup result for executing the backup task, so that a subsequent user can effectively search and access the backup task which is completed. In the embodiment of the application, the data service platform generates the database backup record by using the backup result, and stores the database backup record in the backup log table, so that the record of the information in the execution process of each backup task is realized, and a basis is provided for the follow-up identification of whether the backup is initiated on the same backup database. Therefore, the data service platform can inquire backup results and related information of the execution of a plurality of backup tasks by inquiring the backup log table, so that the data service platform can maintain and manage the database cluster according to the backup log table.

The method provided by the embodiment of the application comprises the steps of acquiring a backup task to be executed, determining a target main database to be backed up indicated by the backup task to be executed, inquiring metadata information of a database cluster in which the target main database is located, and determining a plurality of backup databases in association with the target main database in the database cluster according to a database topological structure included in the metadata information, wherein the database topological structure indicates the association between the plurality of main databases and the plurality of backup databases included in the database cluster, determining the target backup database in the plurality of backup databases, executing data backup operation on the target main database based on the target backup database to obtain a backup result, generating a database backup record comprising the backup result, and storing the database backup record in a backup log table.

Further, as a refinement and extension of the foregoing embodiment, in order to fully describe a specific implementation procedure of the embodiment, an embodiment of the present application provides another data backup method, as shown in fig. 2A, where the method includes:

201. and determining at least one main database in the database cluster to be backed up, and generating a backup task to be executed carrying the main database identification of the main database for each main database.

The active and standby modes of the Redis cluster have better flexibility and can also support larger capacity. However, when the Redis cluster performs data backup, each database node needs to determine the master-slave role of the database node, and the backup is initiated only when the backup database is confirmed, which results in long backup time and easy confusion. In the embodiment of the application, a data service platform determines a database cluster to be backed up, and acquires metadata information of the database cluster, wherein the metadata information comprises a database topological structure and attribute information of each database in the database cluster. Then, the data service platform determines a plurality of databases included in the database cluster, and inquires attribute information corresponding to each database in the plurality of databases in metadata information, wherein the attribute information indicates whether each database in the database cluster to be backed up is a main database or a backup database. And then, at least one main database is extracted from the plurality of databases according to the attribute information so as to generate backup tasks to be executed for each main database later, and the backup operation of the database cluster is orderly completed through the backup tasks to be executed. And then, the data service platform reads the main database identifier of each main database in at least one main database in the metadata information of the database cluster, acquires a preset backup time interval, and can quickly determine which main database needs to be backed up according to the main database identifier, thereby improving the data backup speed of the database cluster. And finally, the data service platform generates backup tasks to be executed, which carry the main database identifiers of the main databases, for each main database at intervals of preset backup time intervals. The operation and maintenance platform can periodically issue backup tasks by deploying configuration timing tasks on the front page and utilizing preset backup time intervals, for example, backup can be initiated every two hours, so that the data information of the database cluster can be better protected, and the high availability of the Redis cluster is ensured.

In an alternative embodiment, when the backup task is deployed by Crontab, once a data switch occurs, the data service platform has difficulty maintaining and adjusting the backup task in the case of multiple backup databases. Therefore, the application remotely initiates the backup task through Celery (distributed task queue), and can execute the backup task asynchronously and concurrently when a plurality of backup tasks occur. Celery supports a plurality of different computers to execute different tasks or the same task, is simple and convenient to use and convenient to maintain, and can realize asynchronous processing of backup tasks. Therefore, when the number of backup tasks to be executed is multiple, the data service platform can form the backup tasks to be executed into a backup task queue to be executed, and call the asynchronous execution thread, so that the backup tasks to be executed are executed asynchronously, the backup tasks are not affected mutually, and the backup speed of the database cluster is improved.

202. And acquiring a backup task to be executed, and determining a target main database to be backed up indicated by the backup task to be executed.

The data backup can enhance the protection of the data and improve the safety of the data, so that the data in the Redis cluster needs to be backed up regularly, and the main and standby modes of the Redis cluster are distributed to different Redis examples through read-write operations, thereby achieving the purpose of improving the data throughput of the database system and enabling the database system to run more stably. Further, in order to enable the Redis cluster to orderly perform data backup, the application determines the main databases to be backed up through the backup task to be executed, does not need to judge the attribute information of each database, and saves the storage space while improving the speed of data backup. In the embodiment of the application, the data service platform acquires the backup task to be executed of the database cluster, determines the target main database to be backed up indicated by the backup task to be executed so as to select the target backup database for the target main database to be backed up later, and completes the backup operation through the backup task to be executed without deploying the backup task at each database node, thereby improving the data backup speed of the database cluster. When the number of the backup tasks to be executed is plural, the backup tasks to be executed acquired by the data service platform are any backup tasks to be executed in the backup task queue to be executed because the backup tasks to be executed are executed asynchronously. Therefore, the backup rate of the database cluster can be improved, and the orderly and stable operation of the database system is ensured.

203. Inquiring metadata information of a database cluster where a target main database is located, and determining a plurality of backup databases in association relation with the target main database in the database cluster according to a database topological structure included in the metadata information.

The metadata information is data describing data attributes, timeliness of the metadata information is good, and regular updating can be performed, so that even if active-standby switching occurs in a Redis cluster, a data service platform can acquire the attribute information of a changed Redis instance in the database topological structure of the metadata information through maintaining the database topological structure of the metadata information, and the target backup database can be accurately determined to execute data backup operation. In the embodiment of the application, a data service platform queries a database cluster in which a target main database is located, and determines a plurality of backup databases in association with the target main database in the database cluster according to a database topology structure included in metadata information of the database cluster, wherein the database topology structure indicates association between the plurality of main databases and the plurality of backup databases included in the database cluster. Therefore, the data service platform can quickly acquire a plurality of backup databases associated with the target main database, and can determine the information of the plurality of backup databases by utilizing the metadata information so as to subsequently determine the target backup database for executing the backup operation, thereby realizing the accurate backup of the data backup.

204. A target backup database is determined among the plurality of backup databases.

If each backup database performs data backup, not only the backup data information is the same, but also a large amount of disk space is occupied by the same data information, so that in order to avoid each backup database executing backup operation, one backup database is selected from a plurality of backup databases as a target backup database to execute backup operation, and the data storage space is saved. In the embodiment of the application, the data service platform selects a backup database from a plurality of backup databases as a target backup database, wherein the target backup database is currently in an idle state or the current residual data storage space is larger than or equal to the data quantity to be backed up of the target main database. In this way, the data service platform can determine a target backup database for the target main database indicated by the backup task to be executed, so that the target backup database executes backup operation on the target main database, and the maintenance and adjustment difficulty of the backup task to be executed can be reduced by considering the running states of the backup databases, thereby improving the accuracy of data backup.

205. And executing data backup operation on the target main database based on the target backup database to obtain a backup result.

Because Redis is an in-memory database and the data volume is not large, the application performs physical backup by using RDB (Relational Database ) files which are native to Redis and can be generally completed within seconds. In the embodiment of the application, the data service platform reads the address information of the main database of the target main database, such as the IP (Intellectual Property, internet protocol) address of the main database, in the metadata information, and reads the address information of the backup database of the target backup database, such as the IP address of the target backup database, in the metadata information. And then, the data service platform controls the target backup database to establish connection with the target main database according to the main database address information and the backup database address information so as to execute data backup operation on the target main database based on the target backup database. And then, acquiring a preset backup script, calling the preset backup script to extract data to be backed up from the target main database, and storing the data to be backed up into a distributed storage space of the target backup database based on the preset backup script. The preset backup script can be a Shell (execution file composed of Shell commands) backup script, and the data backup can be quickly and completely realized by calling the Shell backup script, so that the data backup speed is improved. And finally, when the completion of the execution of the data backup operation is detected, the data service platform calls a preset backup script to acquire a backup result of the data backup operation, wherein the backup result indicates the success or failure of the execution of the data backup operation. In this way, the data service platform enables the target backup database to execute backup operation on the target main database by calling the preset backup script, and meanwhile, the result of the data backup operation is obtained, so that the backup task can be managed and adjusted.

In an alternative embodiment, when the backup operation fails to be executed, a maintenance personnel is required to intervene and identify the cause of the specific failure, so as to repair the cause of the failure and restore the database system to stably run. Therefore, when the backup result indicates that the execution of the data backup operation fails, the data service platform acquires the main database identifier of the target main database and the backup database identifier of the target backup database, and generates a backup failure reminder carrying the main database identifier and the backup database identifier, so that maintenance personnel can determine the main database and the backup database which are abnormal in the database cluster through the main database identifier and the backup database identifier. And then, the data service platform sends the backup failure prompt to terminal equipment where the maintenance personnel are located, so that the maintenance personnel repair the target main database and the target backup database according to the backup failure prompt. When the data service platform receives a data backup instruction initiated after repair of a maintenance person is completed, for example, the maintenance person clicks an immediate execution button after repair to send the data backup instruction, so that backup is triggered once manually. The data service platform acquires a main database identifier and a backup database identifier indicated by the data backup instruction, and determines a target main database indicated by the main database identifier and a target backup database indicated by the backup database identifier. And then, re-executing data backup operation on the target main database based on the target backup database to obtain a backup result. It should be noted that, after the repair personnel completes the repair, the database cluster may trigger the next backup task through the timing task.

206. A database backup record is generated that includes the backup result and is stored in a backup log table.

In order to facilitate maintenance and management of the backup task, the data service platform generates a database backup record including the backup result after obtaining the backup result, so as to record the backup result of the backup task. In the embodiment of the application, the data service platform acquires the backup completion time when the data backup operation is completed, and acquires the backup database identifier of the target backup database and the main database identifier of the target main database in the metadata information. The backup databases and the master databases in the database cluster can be determined by the backup database identification and the master database identification so as to record the backup operation of which backup database is executed on which master database in the backup task. Then, a database backup record including the backup completion time, the backup database identification, the primary database identification, the backup result is generated. And then, the data service platform acquires a backup log table of the database cluster, and stores the database backup record in the backup log table.

In an alternative embodiment, after the backup task is finished, the result of the backup task is displayed on the front page, whether successful or failed, so that the user can acquire the detailed information of the backup task by looking at the page. Specifically, the data service platform obtains a plurality of database backup records included in the backup log table. Then, for each of the plurality of database backup records, record details are generated from the content included in the database backup record, and a primary database identification is extracted from the database backup record, and the record details are associated with the primary database identification. Thus, the detailed information page at the front end can display the backup task indicated by the main database according to the main database identification, so that the layout of the detailed information page is tidy. The data service platform then obtains a plurality of record details of the plurality of database backup records, wherein each record detail of the plurality of record details is associated with a primary database identifier. And finally, the data service platform generates a detailed information page, adds a plurality of pieces of recorded detailed information to the detailed information page, and pushes the detailed information page to the terminal when detecting that the terminal requests to display the detailed information page, so that staff can know the backup information of the database cluster more timely and in detail.

In an alternative embodiment, the backup data that has been backed up is maintained in the distributed storage space of the backup database for a predetermined period of time and then transferred to the storage medium of the backup database. Specifically, the data service platform takes the time point when the backup result is obtained as the backup completion time point, and obtains a preset dump time interval. Then, when the time interval between the current time and the backup completion time point is detected to reach the preset dump time interval, for example, when the time interval between the current time and the backup completion time point is detected to reach 180 days, the data service platform acquires backup data of which backup is completed in the distributed storage space of the target backup database. And then, the backup data is transferred to a database storage medium of the target backup database, and the backup data is kept for a longer period, and meanwhile, the cost is reduced.

In summary, the data backup method provided by the embodiment of the application comprises the following steps:

as shown in fig. 2B, in combination with metadata information of the Redis cluster, it can be determined that the topology of the Redis cluster, for example, s1_1, s1_2, s1_3 are backup nodes of the primary database M1, s2_1, s2_2, s2_3 are backup nodes of the primary database M2, and s3_1, s3_2, s3_3 are backup nodes of the primary database M3. Therefore, after a backup task is remotely initiated through a Celery background task, then before each backup, the data service platform firstly inquires metadata information to acquire backup database information in the Redis cluster, and finally data backup operation is executed based on the backup database. For example, the data service platform selects s1_1 to perform a backup operation on M1, selects s2_3 to perform a backup operation on M2, and selects s3_2 to perform a backup operation on M3 by querying metadata information. The metadata information records the role information of each Redis instance and the primary database or backup database information corresponding to the instance.

Further, as a specific implementation of the method shown in fig. 1, an embodiment of the present application provides a data backup device, as shown in fig. 3A, where the device includes: the system comprises an acquisition module 301, a query module 302, a backup module 303 and a storage module 304.

The obtaining module 301 is configured to obtain a backup task to be executed, and determine a target primary database to be backed up indicated by the backup task to be executed;

the query module 302 is configured to query metadata information of a database cluster where the target primary database is located, and determine, in the database cluster, a plurality of backup databases in association with the target primary database according to a database topology structure included in the metadata information, where the database topology structure indicates association between a plurality of primary databases and a plurality of backup databases included in the database cluster;

a backup module 303, configured to determine a target backup database from the multiple backup databases, and perform a data backup operation on the target primary database based on the target backup database to obtain a backup result;

the storage module 304 is configured to generate a database backup record including the backup result, and store the database backup record in a backup log table.

In a specific application scenario, the backup module 303 is configured to select a backup database from the plurality of backup databases as the target backup database, where the target backup database is a backup database currently in an idle state or a current remaining data storage space is greater than or equal to a data amount to be backed up of the target main database; reading main database address information of the target main database in the metadata information, and reading backup database address information of the target backup database in the metadata information; controlling the target backup database to establish connection with the target main database according to the main database address information and the backup database address information; acquiring a preset backup script, calling the preset backup script to extract data to be backed up from the target main database, and storing the data to be backed up into a distributed storage space of the target backup database based on the preset backup script; and when the completion of the execution of the data backup operation is detected, calling the preset backup script to acquire a backup result of the data backup operation, wherein the backup result indicates the success or failure of the execution of the data backup operation.

In a specific application scenario, the storage module 304 is configured to obtain a backup completion time when the data backup operation is performed, and obtain a backup database identifier of the target backup database and a master database identifier of the target master database in the metadata information; generating a database backup record comprising the backup completion time, the backup database identifier, the main database identifier and the backup result; and acquiring a backup log table of the database cluster, and storing the database backup record in the backup log table.

In a specific application scenario, as shown in fig. 3B, the apparatus further includes: the system comprises a generation module 305, a restoration module 306, a display module 307 and a transfer module 308.

A generating module 305, configured to determine the database cluster to be backed up, and obtain metadata information of the database cluster; determining a plurality of databases included in the database cluster, inquiring attribute information corresponding to each database in the plurality of databases in the metadata information, and extracting at least one main database from the plurality of databases according to the attribute information; reading a main database identifier of each main database in the at least one main database in the metadata information, acquiring a preset backup time interval, and generating a backup task to be executed carrying the main database identifier of the main database for each main database at intervals of the preset backup time interval; when the number of the backup tasks to be executed is multiple, the backup tasks to be executed are formed into a backup task queue to be executed, and an asynchronous execution thread is called, so that the backup tasks to be executed are executed asynchronously.

The repairing module 306 is configured to obtain a main database identifier of the target main database and a backup database identifier of the target backup database when the backup result indicates that the current data backup operation fails to be executed, and generate a backup failure reminder carrying the main database identifier and the backup database identifier; sending the backup failure prompt to terminal equipment where a maintainer is located, so that the maintainer repairs the target main database and the target backup database according to the backup failure prompt; when a data backup instruction initiated after repair of the maintenance personnel is completed is received, acquiring the main database identifier and the backup database identifier indicated by the data backup instruction, determining a target main database indicated by the main database identifier and a target backup database indicated by the backup database identifier, and re-executing data backup operation on the target main database based on the target backup database to obtain a backup result.

A display module 307, configured to obtain a plurality of database backup records included in the backup log table; generating record detailed information according to the content included in the database backup records for each database backup record in the plurality of database backup records, extracting a main database identifier from the database backup records, and associating the record detailed information with the main database identifier; acquiring a plurality of record detailed information of the database backup records, wherein each record detailed information in the plurality of record detailed information is associated with a main database identifier; generating a detailed information page, adding the plurality of recorded detailed information to the detailed information page, and pushing the detailed information page to a terminal when detecting that the terminal requests to display the detailed information page.

The dump module 308 is configured to take a time point at which the backup result is obtained as a backup completion time point, and obtain a preset dump time interval; when the time interval between the current time and the backup completion time point reaches the preset dump time interval, acquiring backup data which is completed in the distributed storage space of the target backup database; and the backup data are transferred to a database storage medium of the target backup database.

The device provided by the embodiment of the application acquires the backup task to be executed, determines the target main database to be backed up indicated by the backup task to be executed, inquires the metadata information of the database cluster in which the target main database is positioned, and determines a plurality of backup databases in association with the target main database in the database cluster according to the database topology structure included in the metadata information, wherein the database topology structure indicates the association between the plurality of main databases and the plurality of backup databases included in the database cluster, determines the target backup database in the plurality of backup databases, executes data backup operation on the target main database based on the target backup database to obtain a backup result, generates a database backup record comprising the backup result, and stores the database backup record in a backup log table, and inquires the number of backup databases in the cluster through the metadata information and acquires one backup database information.

It should be noted that, for other corresponding descriptions of each functional unit related to the data backup device provided by the embodiment of the present application, reference may be made to corresponding descriptions in fig. 1 and fig. 2A to fig. 2B, and no further description is given here.

In an exemplary embodiment, referring to fig. 4, there is also provided a computer device, which includes a bus, a processor, a memory, and a communication interface, and may further include an input-output interface and a display device, where each functional unit may perform communication with each other through the bus. The memory stores a computer program and a processor for executing the program stored in the memory to perform the data backup method in the above embodiment.

A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the data backup method.

From the above description of the embodiments, it will be clear to those skilled in the art that the present application may be implemented in hardware, or may be implemented by means of software plus necessary general hardware platforms. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.), and includes several instructions for causing a computer device (may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective implementation scenario of the present application.

Those skilled in the art will appreciate that the drawing is merely a schematic illustration of a preferred implementation scenario and that the modules or flows in the drawing are not necessarily required to practice the application.

Those skilled in the art will appreciate that modules in an apparatus in an implementation scenario may be distributed in an apparatus in an implementation scenario according to an implementation scenario description, or that corresponding changes may be located in one or more apparatuses different from the implementation scenario. The modules of the implementation scenario may be combined into one module, or may be further split into a plurality of sub-modules.

The above-mentioned inventive sequence numbers are merely for description and do not represent advantages or disadvantages of the implementation scenario.

The foregoing disclosure is merely illustrative of some embodiments of the application, and the application is not limited thereto, as modifications may be made by those skilled in the art without departing from the scope of the application.

Claims

1. A method of data backup, comprising:

2. The method of claim 1, wherein the obtaining the backup task to be performed, before determining the target primary database to be backed up indicated by the backup task to be performed, further comprises:

3. The method of claim 1, wherein determining a target backup database among the plurality of backup databases, performing a data backup operation on the target primary database based on the target backup database, and obtaining a backup result, comprises:

4. The method of claim 1, wherein the determining a target backup database from the plurality of backup databases performs a data backup operation on the target primary database based on the target backup database, and wherein after obtaining the backup result, the method further comprises:

5. The method of claim 1, wherein the generating a database backup record comprising the backup result and storing the database backup record in a backup log table comprises:

6. The method according to claim 1, wherein the method further comprises:

7. The method according to claim 1, wherein the method further comprises:

8. A data backup apparatus, comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.