CN117149527B - System and method for backing up and recovering server data - Google Patents
System and method for backing up and recovering server data Download PDFInfo
- Publication number
- CN117149527B CN117149527B CN202311429364.7A CN202311429364A CN117149527B CN 117149527 B CN117149527 B CN 117149527B CN 202311429364 A CN202311429364 A CN 202311429364A CN 117149527 B CN117149527 B CN 117149527B
- Authority
- CN
- China
- Prior art keywords
- backup
- data
- server
- frequency
- recovery
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000011084 recovery Methods 0.000 claims abstract description 48
- 230000005540 biological transmission Effects 0.000 claims description 11
- 238000013138 pruning Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 230000035945 sensitivity Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 230000005484 gravity Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 15
- 230000008569 process Effects 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011980 disaster recovery test Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a system and a method for backing up and recovering server data, which relate to the technical field of data backup and recovery, and particularly relates to a system and a method for backing up and recovering server data, wherein the system and the method collect and analyze backup data of a server and classify the backup data according to types; evaluating the importance of the backup data, and carrying out higher-frequency backup on the important backup data, otherwise, reducing the backup frequency; acquiring the update frequency of backup data, adopting higher-frequency incremental backup for frequently-changed data, and adopting full-quantity backup or low-frequency incremental backup for less-changed data; after all backup data are collected, identifying and removing redundant data; and after the data recovery instruction is acquired, carrying out data recovery. The invention adopts an intelligent data backup strategy optimization algorithm, flexibly adjusts the backup plan according to factors such as data importance, update frequency and the like, effectively improves the data backup efficiency, and reduces the backup time and the resource occupation.
Description
Technical Field
The invention relates to the technical field of data backup and recovery, in particular to a server data backup and recovery system and a server data backup and recovery method.
Background
In the present digital age, backup and recovery of server data is becoming critical. The server stores a large amount of important data including enterprise data, user information, business data, etc. Once a server fails, data is lost or suffers from malicious attack, the service operation and data security will be seriously affected. Therefore, it becomes important to realize an efficient and reliable server data backup and recovery system.
Currently, there are server data backup and restore systems on the market that use conventional backup techniques, such as full back-up and incremental back-up. However, these conventional backup methods have some problems in that the backup speed is slow: the conventional backup method requires a long time when backing up a large amount of data, resulting in a long time consuming backup process. For example, the storage space occupation is large: full and incremental backups occupy a large amount of storage space, increasing the cost of data backups. The recovery speed is slow: when faults occur, the data recovery speed of the traditional backup method is low, and the quick recovery of the service is affected. Backup policy is fixed: the backup strategy in the existing system is usually fixed and cannot be adaptively adjusted, so that the backup efficiency is low. Privacy and security risks: in the data backup process, hidden danger of data leakage and unauthorized access may exist, and effective privacy protection measures are lacked.
Disclosure of Invention
The present invention is directed to a system and method for backing up and recovering server data, which solve the above-mentioned problems in the prior art.
The embodiment of the invention is realized by the following technical scheme:
in a first aspect, the present invention provides a method for backing up and recovering server data, including;
collecting and analyzing backup data of the server, and classifying the backup data according to types;
evaluating the importance of the backup data, and carrying out higher-frequency backup on the important backup data, otherwise, reducing the backup frequency;
acquiring the update frequency of backup data, adopting incremental backup of a preset high-frequency value for frequently changed data, and adopting full-scale backup or incremental backup of a preset low-frequency value for less changed data;
after all backup data are collected, identifying and removing redundant data;
after acquiring a data recovery instruction, carrying out data recovery;
wherein, frequent changes: is higher than a preset high-frequency judgment threshold value; less variation: the current data change frequency is lower than a preset low-frequency judgment threshold value.
In one embodiment of the present invention, the performing data recovery includes;
setting a plurality of backup points and recording difference data between each backup point;
when data recovery is needed, selecting a proper backup point according to the need, and recovering the difference data between the backup point and the previous backup point;
wherein the appropriate backup point: the server with the highest comprehensive score in all the servers comprises network bandwidth, transmission delay and storage space.
In one embodiment of the present invention, further comprising;
setting corresponding backup frequency of the backup task, and backing up the corresponding backup task according to the backup frequency;
setting the importance and the emergency degree of the backup data, and determining the priority of the backup data according to the importance and the emergency degree of the backup data;
selecting full backup and incremental backup according to the priority of the backup data;
monitoring the execution condition of the backup task and judging whether the backup task is successful or not;
if the backup fails, a backup failure signal is sent out, and the backup is carried out again.
In one embodiment of the present invention, further comprising;
a buffer area is arranged between the backup points and is used for storing difference data;
and when one of the two adjacent backup points is abnormal, the difference data stored in the buffer area is resent to the next adjacent backup point, and when the difference data is written into the next buffer area, the difference data stored in the last buffer area is deleted.
In one embodiment of the present invention, further comprising;
setting a first server and a second server, backing up data backup in the first server and the second server respectively, and synchronizing the first server and the second server in real time;
the first server is used for normal task processing, and the second server only collects backup data;
and when the first server fails, switching to a second server.
In an embodiment of the present invention, the real-time synchronization between the first server and the second server includes;
setting a master node and a slave node, and synchronizing the master node and the slave node through full synchronization or incremental synchronization;
if the slave node does not respond, judging whether the slave node is temporarily unresponsive or completely unresponsive, and selecting to use full synchronization or incremental synchronization;
acquiring the identification of a queue storage master node and the offset of current transmission, and if the identification of the queue storage master node and the offset of the current transmission are both in the queue when the current slave node is reconnected to the system, judging that the slave node is temporarily unresponsive, and continuing to transmit data from the last relay position;
if the identification of the queue storage master node is inconsistent with that of the previous slave node when the master node is reconnected to the system, full synchronization is adopted;
if the identification of the queue storage master node is inconsistent with that of the previous slave node when the system is reconnected, but the offset does not exist, full synchronization is adopted;
the rest uses incremental synchronization.
In one embodiment of the present invention, the identifying and removing redundant data includes;
removing a part of redundant data from the original data set to obtain a core data set;
performing local model training according to the core data set to obtain local model updating;
the local model is thinned through the pruning rate of the model iterated in n rounds;
uploading the thinned local model to a cloud server, and after receiving the thinned local model, carrying out global model aggregation by the cloud server to obtain an updated global model;
and the cloud server performs secondary pruning on the more-row global model, and redundant data is removed through the global model after secondary pruning.
In one embodiment of the present invention, the obtaining the core dataset includes;
setting the tolerance of the loss function of the core data set, and defining the tolerance as the influence on the global model loss function when only the core data set of one device is used for model training;
defining the sensitivity of each data point as the specific gravity of the loss function generated by that point relative to the weighted average loss function of all data points;
the core dataset is built by tolerance and sensitivity.
In a second aspect, the present invention provides a server data backup and restore system, comprising
The collection module is configured to collect and analyze backup data of the server and classify the backup data according to types;
the backup module is configured to evaluate the importance of the backup data, and to backup the important backup data with higher frequency, otherwise, the backup frequency is reduced;
the updating module is configured to acquire the updating frequency of the backup data, and for frequently changed data, the updating module adopts incremental backup of a preset high-frequency value, and for less changed data, the updating module adopts full-scale backup or incremental backup of a preset low-frequency value;
the removing module is configured to collect all backup data, identify and remove redundant data;
the recovery module is configured to recover the data after acquiring the data recovery instruction;
the main control module is connected with the collecting module, the backup module, the updating module, the removing module and the recovering module and is used for executing the server data backup and recovering method.
In a third aspect, the present invention further provides a computer readable storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements a server data backup and recovery method as described above.
The technical scheme of the embodiment of the invention has at least the following advantages and beneficial effects:
the data backup efficiency of the invention is higher: the invention adopts an intelligent data backup strategy optimization algorithm, flexibly adjusts the backup plan according to factors such as data importance, update frequency and the like, effectively improves the data backup efficiency, and reduces the backup time and the resource occupation.
The data recovery speed is faster: the invention adopts the incremental backup and differential backup technology to support the rapid data recovery, only needs to recover partial changed data, reduces the time required by data recovery and improves the fault recovery speed of the server.
Automatic backup scheduling is more intelligent: the intelligent backup scheduling system is introduced, so that the running state and the load condition of the server can be monitored in real time, the backup plan can be dynamically adjusted according to the actual condition, and unnecessary influence of the backup task on the performance of the server is avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be executed according to the time/logic sequence indicated by the naming or numbering, and the execution sequence of the steps in the flow that are named or numbered may be changed according to the technical purpose to be achieved, so long as the same or similar technical effects can be achieved.
The division of the modules presented in this application is a logical division, and there may be other manners of division in practical application, for example, multiple modules may be combined or integrated in another system, or some features may be omitted, or not performed.
The modules or sub-modules described separately may or may not be physically separate, may or may not be implemented in software, and may be implemented in part in software, where the processor invokes the software to implement the functions of the part of the modules or sub-modules, and where other parts of the templates or sub-modules are implemented in hardware, for example in hardware circuits. In addition, some or all of the modules may be selected according to actual needs to achieve the purposes of the present application.
Referring to fig. 1, the method for backing up and recovering server data provided by the present invention includes;
server backup history and behavior patterns are learned by AI algorithm. The system collects and analyzes the backup data of the server, including backup frequency, backup data amount, backup file type, etc., to build a model of the backup data.
S101: collecting and analyzing backup data of the server, and classifying the backup data according to types;
such as databases, configuration files, logs, etc. The requirements and the frequency of the backup of different types of data can be different, and the backup requirements of different data types can be better adapted through classification.
S102: evaluating the importance of the backup data, and carrying out higher-frequency backup on the important backup data, otherwise, reducing the backup frequency;
for important data, the backup frequency may be higher to ensure timely backup and restoration of the data. While for less important data, the backup frequency may be reduced appropriately to reduce backup overhead.
S103: acquiring the update frequency of backup data, adopting incremental backup of a preset high-frequency value for frequently changed data, and adopting full-scale backup or incremental backup of a preset low-frequency value for less changed data;
for frequently changing data, an incremental backup of a preset high frequency value may be taken, while for less changing data, a full-scale backup or an incremental backup of a preset low frequency value may be taken.
Based on the above learning and analysis, the system dynamically adjusts the backup strategy. The adjustment of the backup strategy includes backup frequency, backup level (full backup, incremental backup), differential backup, etc. These adjustments are all made automatically without manual intervention by an administrator.
S104: after all backup data are collected, identifying and removing redundant data;
s105: after acquiring a data recovery instruction, carrying out data recovery;
wherein, frequent changes: is higher than a preset high-frequency judgment threshold value; less variation: the current data change frequency is lower than a preset low-frequency judgment threshold value.
According to an exemplary embodiment of the present invention, the performing data recovery includes;
setting a plurality of backup points and recording difference data between each backup point; when data recovery is needed, selecting a proper backup point according to the need, and recovering the difference data between the backup point and the previous backup point;
wherein the appropriate backup point: the server with the highest comprehensive score in all the servers comprises network bandwidth, transmission delay and storage space.
Specifically, the recovery speed is further improved, and the system can adopt a multi-backup combined method. That is, during the backup process, the system may generate a plurality of backup points and record the difference data between each backup point. When the data is recovered, a proper backup point can be selected for recovery according to the requirement, so that a faster recovery speed is realized;
wherein the appropriate backup point: the server with the highest comprehensive score in all the servers comprises network bandwidth, transmission delay and storage space.
Data compression and acceleration: the system of the invention supports the compression of the backup data to reduce the size of the backup data. When the data is restored, the system can automatically decompress the backup data, thereby ensuring the high efficiency of the restoration process.
Parallel recovery: to further increase recovery speed, the system of the present invention supports parallel recovery. The data of a plurality of backup nodes can be restored simultaneously, the server resources are fully utilized, and the speed of data restoration is increased.
By the aid of the technical scheme of recovery speed optimization, the system can achieve faster and more efficient data recovery. The user can recover the lost data in a short time, reduce the service interruption time and improve the availability and stability of the system.
An exemplary embodiment of the present invention, further comprising; setting corresponding backup frequency of the backup task, and backing up the corresponding backup task according to the backup frequency; setting the importance and the emergency degree of the backup data, and determining the priority of the backup data according to the importance and the emergency degree of the backup data; selecting full backup and incremental backup according to the priority of the backup data; monitoring the execution condition of the backup task and judging whether the backup task is successful or not; if the backup fails, a backup failure signal is sent out, and the backup is carried out again.
Specifically, the user sets a backup task including a backup frequency, a backup target, a backup data type, and the like. The user can flexibly set different types of backup tasks according to actual demands.
Once the backup task setting is completed, the system will automatically schedule the backup tasks according to the backup frequency set by the user. The system automatically triggers the execution of the backup task according to a predetermined time period.
For a plurality of backup tasks, the system can automatically adjust the backup priority according to the importance and the emergency degree of the backup data. The priority of backup of important data is higher, so that backup can be performed earlier, and the data safety is ensured.
The system of the invention supports a multi-level backup strategy, namely, backups of different levels can be set. For example, full backup and incremental backup can be set at the same time, and the backup mode is automatically selected according to the actual situation, so that the backup data volume and the backup time are reduced.
During the backup process, the system monitors the execution of the backup task. If the backup task is found to fail or be wrong, the system can send an alarm to inform an administrator in time and try to re-execute the backup task.
The system can record the execution log of the backup task in detail, including the information of the backup start time, the backup end time, the backup data volume and the like. An administrator can check the backup log at any time to know the execution condition of the backup task.
Through the technical scheme of automatic backup scheduling, the system can realize automatic management of the backup tasks, reduce the workload of manual intervention of an administrator and ensure on-time execution of the backup tasks. Meanwhile, the adjustment of the backup priority and the application of the multi-stage backup strategy can better meet different demands of users on backup, and the flexibility and reliability of backup tasks are improved.
In an exemplary embodiment of the present invention, since two backup points may have errors when transferring data information, the system receives the information of data transmission failure and resubmits the data to a new backup point. When the wrong backup point is re-started and then re-added to the backup node unit network with a new identity, the situation that the data is submitted twice in the backup occurs, and the data between the two backup points is inconsistent.
Therefore, a buffer area is arranged between the backup points and the backup points, and the buffer area is used for storing difference data; and when one of the two adjacent backup points is abnormal, the difference data stored in the buffer area is resent to the next adjacent backup point, and when the difference data is written into the next buffer area, the difference data stored in the last buffer area is deleted.
An exemplary embodiment of the present invention, further comprising;
setting a first server and a second server, backing up data backup in the first server and the second server respectively, and synchronizing the first server and the second server in real time; the first server is used for normal task processing, and the second server only collects backup data; and when the first server fails, switching to a second server.
Specifically, the data is backed up to a plurality of independent storage devices, so that the backup of the data is ensured to have a plurality of copies. When the main storage device fails, the main storage device can be quickly switched to the standby storage device, so that seamless switching and continuous availability of data are realized.
The system is provided with a fault detection mechanism, so that fault conditions of server hardware, a network and the like can be monitored in real time. Once a fault is found, the system automatically triggers a fault recovery mechanism and automatically switches to the standby equipment, ensuring high availability of the system.
When the main server fails, the system automatically switches to the standby server. In the disaster recovery switching process, the system can realize seamless switching of data and ensure continuous operation of the service.
When faults or data loss occur, the system can realize quick recovery. By backing up data in advance, adopting incremental backup and differential backup techniques, etc., the data can be recovered in a short time, and the service interruption time is reduced.
In order to further improve the reliability of the system, the system of the invention can adopt a dual-machine hot standby technology. The standby server and the main server keep real-time synchronization, so that the standby server is ready to take over the work of the main server at any time, and the dual-machine hot standby with high reliability is realized.
In order to ensure the effectiveness of disaster recovery and high reliability design, the system of the invention supports a disaster recovery test function. And periodically performing disaster tolerance test, verifying the reliability of backup equipment and a recovery mechanism, and timely finding and solving potential problems.
Through the technical scheme of disaster recovery and high-reliability design, the system can ensure continuous availability of server data, improve stability and reliability of the server system, and meet requirements of users on high availability and high reliability.
According to an exemplary embodiment of the present invention, the first server and the second server are synchronized in real time, including; the master node and the slave node are set, and the master node and the slave node are synchronized through full-volume synchronization or incremental synchronization, but if the slave node does not respond temporarily, direct use of full-volume synchronization causes additional overhead of the system, so the following steps provide a method for selecting proper volume synchronization or incremental synchronization at proper timing for synchronization.
S201: if the slave node does not respond, judging whether the slave node is temporarily unresponsive or completely unresponsive, and selecting to use full synchronization or incremental synchronization;
s202: acquiring the identification of a queue storage master node and the offset of current transmission, and if the identification of the queue storage master node and the offset of the current transmission are both in the queue when the current slave node is reconnected to the system, judging that the slave node is temporarily unresponsive, and continuing to transmit data from the last relay position;
s203: if the identification of the queue storage master node is inconsistent with that of the previous slave node when the master node is reconnected to the system, full synchronization is adopted;
s204: if the identification of the queue storage master node is inconsistent with that of the previous slave node when the master node is reconnected to the system, but the offset does not exist, full synchronization is adopted, and incremental synchronization is adopted in the rest cases.
In an exemplary embodiment of the present invention, the identifying and removing redundant data includes;
s301: the method comprises the steps of obtaining an original data set, and removing partial redundant data according to the original data set to obtain a core data set;
s302: performing local model training according to the core data set to obtain local model updating;
s303: the updated local model is thinned through the model pruning rate of n rounds of iteration;
s304: uploading the thinned local model to a cloud server, and after receiving the thinned local model, carrying out global model aggregation by the cloud server to obtain an updated global model;
s305: and the cloud server performs secondary pruning on the more-row global model, and redundant data is removed through the global model after secondary pruning.
Specifically, the obtaining the core data set includes; setting the tolerance of the loss function of the core data set, and defining the tolerance as the influence on the global model loss function when only the core data set of one device is used for model training; defining the sensitivity of each data point as the specific gravity of the loss function generated by that point relative to the weighted average loss function of all data points; the core dataset is built by tolerance and sensitivity.
Defining toleranceIt is assumed that the original dataset of the device can be slave +.>Removing part of redundant data to obtain sub data set +.>Satisfy->And use +.>Model training is performed, and the influence on the global model satisfies the following formula:
≤/>
then call forIs->Is->Proximity.
Wherein 0 < >< 1, indicating device global model loss function tolerance, +.>Representing the global loss function obtained by model training of all devices using the original dataset,/i->Representing the global loss function resulting from model training by only the user of all users using subset i.
Define sensitivity, recordFor dataset +.>The sensitivity of any data point j of the data point j is +.>Can be expressed as:
is indicated at->Effect of data point j on change on model loss function of all data points in data set, +.>The larger the data point j, the more important it should be contained in the core dataset.
To sum up, the device set isThe data set is +.>The global model loss function tolerance is +.>Error rate is +.>Parameter->Number of clusters K.
PartitioningFor K clusters, data point j=1: />；
According toAssessment->The formula is as follows;
the core dataset is provided by the following formula:
in the method, in the process of the invention,for dataset +.>Sensitive boundaries of data point j.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. The computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A server data backup and recovery method is characterized by comprising the following steps of;
collecting and analyzing backup data of the server, and classifying the backup data according to types;
evaluating the importance of the backup data, and carrying out higher-frequency backup on the important backup data, otherwise, reducing the backup frequency;
acquiring the update frequency of backup data, adopting incremental backup of a preset high-frequency value for frequently changed data, and adopting full-scale backup or incremental backup of a preset low-frequency value for less changed data;
after all backup data are collected, identifying and removing redundant data;
after acquiring a data recovery instruction, carrying out data recovery;
wherein, frequent changes: the current data change frequency is higher than a preset high-frequency judgment threshold value; less variation: the current data change frequency is lower than a preset low-frequency judgment threshold value;
setting a first server and a second server, backing up data backup in the first server and the second server respectively, and synchronizing the first server and the second server in real time;
the first server is used for normal task processing, and the second server only collects backup data;
when the first server breaks down, switching to a second server;
the real-time synchronization of the first server and the second server comprises the following steps:
setting a master node and a slave node, and synchronizing the master node and the slave node through full synchronization or incremental synchronization;
if the slave node does not respond, judging whether the slave node is temporarily unresponsive or completely unresponsive, and selecting to use full synchronization or incremental synchronization;
acquiring the identification of a queue storage master node and the offset of current transmission, and if the identification of the queue storage master node and the offset of the current transmission are both in the queue when the current slave node is reconnected to the system, judging that the slave node is temporarily unresponsive, and continuing to transmit data from the last relay position;
if the identification of the queue storage master node is inconsistent with that of the previous slave node when the master node is reconnected to the system, full synchronization is adopted;
if the identification of the queue storage master node is inconsistent with that of the previous slave node when the system is reconnected, but the offset does not exist, full synchronization is adopted;
incremental synchronization is used for the rest of the cases;
the identifying and removing redundant data comprises;
the method comprises the steps of obtaining an original data set, and removing partial redundant data according to the original data set to obtain a core data set;
performing local model training according to the core data set to obtain local model updating;
the updated local model is thinned through the model pruning rate of n rounds of iteration;
uploading the thinned local model to a cloud server, and after receiving the thinned local model, carrying out global model aggregation by the cloud server to obtain an updated global model;
the cloud server performs secondary pruning on the more-row global model, and redundant data is removed through the global model after secondary pruning;
the obtaining a core data set comprises;
setting the tolerance of the loss function of the core data set, and defining the tolerance as the influence on the global model loss function when only the core data set of one device is used for model training;
defining the sensitivity of each data point as the specific gravity of the loss function generated by that point relative to the weighted average loss function of all data points;
the core dataset is built by tolerance and sensitivity.
2. The method for backing up and recovering data on a server according to claim 1, wherein said performing data recovery comprises;
setting a plurality of backup points and recording difference data between each backup point;
when data recovery is needed, selecting a proper backup point according to the need, and recovering the difference data between the backup point and the previous backup point;
wherein the appropriate backup point: the server with the highest comprehensive score in all the servers comprises network bandwidth, transmission delay and storage space.
3. The method for backing up and restoring data on a server according to claim 2, further comprising;
setting corresponding backup frequency of the backup task, and backing up the corresponding backup task according to the backup frequency;
setting the importance and the emergency degree of the backup data, and determining the priority of the backup data according to the importance and the emergency degree of the backup data;
selecting full backup and incremental backup according to the priority of the backup data;
monitoring the execution condition of the backup task and judging whether the backup task is successful or not;
if the backup fails, a backup failure signal is sent out, and the backup is carried out again.
4. A server data backup and restore method according to claim 3, further comprising;
a buffer area is arranged between the backup points and is used for storing difference data;
and when one of the two adjacent backup points is abnormal, the difference data stored in the buffer area is resent to the next adjacent backup point, and when the difference data is written into the next buffer area, the difference data stored in the last buffer area is deleted.
5. A server data backup and recovery system, comprising;
the collection module is configured to collect and analyze backup data of the server and classify the backup data according to types;
the backup module is configured to evaluate the importance of the backup data, and to backup the important backup data with higher frequency, otherwise, the backup frequency is reduced;
the updating module is configured to acquire the updating frequency of the backup data, and for frequently changed data, the updating module adopts incremental backup of a preset high-frequency value, and for less changed data, the updating module adopts full-scale backup or incremental backup of a preset low-frequency value;
the removing module is configured to collect all backup data, identify and remove redundant data;
the recovery module is configured to recover the data after acquiring the data recovery instruction;
the main control module is connected with the collecting module, the backup module, the updating module, the removing module and the recovering module and is used for executing the server data backup and recovering method according to any one of claims 1-4.
6. A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, the computer program when executed by a processor implementing a server data backup and restore method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311429364.7A CN117149527B (en) | 2023-10-31 | 2023-10-31 | System and method for backing up and recovering server data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311429364.7A CN117149527B (en) | 2023-10-31 | 2023-10-31 | System and method for backing up and recovering server data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117149527A CN117149527A (en) | 2023-12-01 |
| CN117149527B true CN117149527B (en) | 2024-03-08 |
Family
ID=88908544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311429364.7A Active CN117149527B (en) | 2023-10-31 | 2023-10-31 | System and method for backing up and recovering server data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149527B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117950703B (en) * | 2024-03-26 | 2024-05-31 | 成都赛力斯科技有限公司 | Self-adaptive updating method and device for dimension data, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617098A (en) * | 2013-12-03 | 2014-03-05 | 上海爱数软件有限公司 | Intelligent backup method and system based on data changes |
| CN106502834A (en) * | 2016-10-25 | 2017-03-15 | 广东欧珀移动通信有限公司 | The backup method of data, apparatus and system |
| CN107315657A (en) * | 2017-06-19 | 2017-11-03 | 浙江大华技术股份有限公司 | A kind of method and device of data backup |
| WO2020207010A1 (en) * | 2019-04-08 | 2020-10-15 | 平安科技（深圳）有限公司 | Data backup method and device, and computer-readable storage medium |
| CN112364913A (en) * | 2020-11-09 | 2021-02-12 | 重庆大学 | Federal learning communication traffic optimization method and system based on core data set |
| CN112650622A (en) * | 2020-12-24 | 2021-04-13 | 江苏云柜网络技术有限公司 | Data backup and recovery system and method suitable for intelligent express cabinet service |
| CN112822256A (en) * | 2020-12-31 | 2021-05-18 | 上海英方软件股份有限公司 | Method and device for transmitting data stream of quotation file |
| CN116841801A (en) * | 2023-07-14 | 2023-10-03 | 广州数源畅联科技有限公司 | Data backup method and system based on cloud server |
-
2023
- 2023-10-31 CN CN202311429364.7A patent/CN117149527B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617098A (en) * | 2013-12-03 | 2014-03-05 | 上海爱数软件有限公司 | Intelligent backup method and system based on data changes |
| CN106502834A (en) * | 2016-10-25 | 2017-03-15 | 广东欧珀移动通信有限公司 | The backup method of data, apparatus and system |
| CN107315657A (en) * | 2017-06-19 | 2017-11-03 | 浙江大华技术股份有限公司 | A kind of method and device of data backup |
| WO2020207010A1 (en) * | 2019-04-08 | 2020-10-15 | 平安科技（深圳）有限公司 | Data backup method and device, and computer-readable storage medium |
| CN112364913A (en) * | 2020-11-09 | 2021-02-12 | 重庆大学 | Federal learning communication traffic optimization method and system based on core data set |
| CN112650622A (en) * | 2020-12-24 | 2021-04-13 | 江苏云柜网络技术有限公司 | Data backup and recovery system and method suitable for intelligent express cabinet service |
| CN112822256A (en) * | 2020-12-31 | 2021-05-18 | 上海英方软件股份有限公司 | Method and device for transmitting data stream of quotation file |
| CN116841801A (en) * | 2023-07-14 | 2023-10-03 | 广州数源畅联科技有限公司 | Data backup method and system based on cloud server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117149527A (en) | 2023-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9785521B2 (en) | Fault tolerant architecture for distributed computing systems | |
| US9773015B2 (en) | Dynamically varying the number of database replicas | |
| US9170888B2 (en) | Methods and apparatus for virtual machine recovery | |
| US7844856B1 (en) | Methods and apparatus for bottleneck processing in a continuous data protection system having journaling | |
| CN117149527B (en) | System and method for backing up and recovering server data | |
| CN107544832B (en) | Method, device and system for monitoring process of virtual machine | |
| CN103763155A (en) | Multi-service heartbeat monitoring method for distributed type cloud storage system | |
| CN105607973B (en) | Method, device and system for processing equipment fault in virtual machine system | |
| CN105335250B (en) | A kind of data reconstruction method and device based on distributed file system | |
| CN109491609B (en) | Cache data processing method, device and equipment and readable storage medium | |
| US10558547B2 (en) | Methods for proactive prediction of disk failure in a RAID group and devices thereof | |
| CN108647118B (en) | Storage cluster-based copy exception recovery method and device and computer equipment | |
| CN109117310A (en) | Realize disaster tolerance system, the method and device of data backup | |
| CN102902615A (en) | Failure alarm method and system for Lustre parallel file system | |
| CN110727508A (en) | Task scheduling system and scheduling method | |
| CN114064374A (en) | Fault detection method and system based on distributed block storage | |
| CN108228678A (en) | A kind of more copy data restoration methods and device | |
| CN111309515B (en) | Disaster recovery control method, device and system | |
| CN116781488A (en) | Database high availability implementation method, device, database architecture, equipment and product | |
| CN100435105C (en) | System regeneration method for application server in cluster environment | |
| CN109474694A (en) | A kind of management-control method and device of the NAS cluster based on SAN storage array | |
| CN113656367A (en) | BMC log collection method and system | |
| CN105487946A (en) | Fault computer automatic switching method and device | |
| CN111737043B (en) | Database disaster recovery method, device, server and storage medium | |
| CN111694894A (en) | Method, server, device and storage medium for monitoring data synchronization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |