CN113987065A - Database drifting method, system, electronic device and storage medium - Google Patents
Database drifting method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN113987065A CN113987065A CN202111143652.7A CN202111143652A CN113987065A CN 113987065 A CN113987065 A CN 113987065A CN 202111143652 A CN202111143652 A CN 202111143652A CN 113987065 A CN113987065 A CN 113987065A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- instance
- load
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database drifting method, a system, electronic equipment and a storage medium, wherein the database drifting method comprises the following steps: detecting whether a first instance loaded with a first database meets a preset high-load condition; when the first instance meets the high load condition, determining data to be drifted in the first database, and creating a second database and a second instance; synchronizing the data to be drifted into the second database and loading the second database to the second instance. The invention monitors the load of each database instance in the production environment, drifts the data of the database to the newly added instance for the instance with higher load, has simple operation process and no need of manual guard, shortens the interruption time of the service, reduces the manual workload and reduces the production environment faults caused by manual operation errors while meeting the requirement of realizing the dynamic balance of the load of the production environment.
Description
Technical Field
The present invention relates to the field of databases, and in particular, to a database drifting method, system, electronic device, and storage medium.
Background
An instance refers to a set of operating system processes (or a multi-threaded process) and some memory, which may operate on a database. The database refers to a file set and comprises a data file, a control file, a temporary file, a redo log file and the like. Where a database can be loaded and opened by multiple instances, an instance can load and open one database at any point in time, more precisely, an instance can load and open at most one database throughout its lifetime.
With the increasing traffic, database instances are prone to load imbalances in the production environment, thereby causing failures. In the prior art, the problem of unbalanced load of a database instance is mainly corrected by a method of manually unloading or creating a copy link, and these methods need to make a more complex process, are time-consuming, cause a longer service interruption time, and require a large amount of manual operations in the whole operation process, cause a larger workload, and easily cause a fault to the production environment due to manual operation errors.
Disclosure of Invention
The technical problem to be solved by the invention is to overcome the defects of long service interruption time, large workload and more faults caused by manually balancing database instance load of a production environment in the prior art, and provide a database drifting method, a database drifting system, electronic equipment and a storage medium.
The invention solves the technical problems through the following technical scheme:
according to a first aspect of the present invention, there is provided a database drifting method comprising the steps of:
detecting whether a first instance loaded with a first database meets a preset high-load condition;
when the first instance meets the high load condition, determining data to be drifted in the first database, and creating a second database and a second instance;
synchronizing the data to be drifted into the second database and loading the second database to the second instance.
Preferably, between the step of creating the second database and the second instance and the step of synchronizing the data to be drifted into the second database, the method further includes:
establishing a first data synchronization link between the second database and the first database so as to synchronize the data to be drifted into the second database through the first data synchronization link.
Preferably, after the step of synchronizing the data to be drifted into the second database, the method further comprises:
checking whether the data synchronized to the second database is consistent with the data to be drifted;
and if the verification is passed, switching the service flow accessing the first database to the second database.
Preferably, the database drifting method further includes:
detecting whether the first instance meets a preset low load condition;
when the first instance meets the low load condition, determining a third instance to be merged from instances other than the first instance, and synchronizing data in a third database loaded by the third instance into the first database, wherein the first instance after synchronization does not meet the high load condition.
Preferably, between the step of determining a third instance to be merged from the instances other than the first instance and the step of synchronizing the data in the third database loaded by the third instance into the first database, further comprising:
establishing a second data synchronization link between the third database and the first database so as to synchronize data in the third database to the first database through the second data synchronization link.
Preferably, after the step of synchronizing the data in the database loaded by the third instance into the first database, the method further comprises:
checking whether the data synchronized to the first database is consistent with the data in the third database;
and if the verification is passed, switching the service flow accessing the third database to the first database.
Preferably, the first instance is spatially cleaned.
According to a second aspect of the present invention, there is provided a database drift system, comprising a load detection module, a first drift preparation module and a first data synchronization module;
the load detection module is used for detecting whether a first instance loaded with a first database meets a preset high load condition;
the first drift preparation module is used for determining data to be drifted in the first database and creating a second database and a second example when the first example meets the high load condition;
the first data synchronization module is used for synchronizing the data to be drifted into the second database and loading the second database to the second instance.
Preferably, the database drift system further comprises a first link construction module;
the first link building module is used for building a first data synchronization link between the second database and the first database so as to synchronize the data to be drifted into the second database through the first data synchronization link.
Preferably, the database drifting system further comprises a first data checking module and a first traffic switching module;
the first data checking module is used for checking whether the data synchronized to the second database is consistent with the data to be drifted;
and the first traffic switching module is used for switching the service traffic accessing the first database to the second database when the check is passed.
Preferably, the load detection module is further configured to detect whether the first instance meets a preset low load condition;
the database drift module further comprises:
a second drift preparation module for determining a third instance to be merged from instances other than the first instance when the first instance satisfies the low load condition;
a second data synchronization module, configured to synchronize data in a third database loaded by the third instance into the first database, where the first instance after synchronization does not satisfy the high load condition.
Preferably, the database drift system further comprises a second link building module;
the second link building module is used for building a second data synchronization link between the third database and the first database so as to synchronize the data in the third database to the first database through the second data synchronization link.
Preferably, the database drifting system further comprises a second data checking module and a second traffic switching module:
the second data checking module is used for checking whether the data synchronized to the first database is consistent with the data in the third database;
and the second traffic switching module is used for switching the service traffic accessing the third database to the first database when the check is passed.
According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor connected to the memory, the processor implementing the database drifting method of the present invention when executing a computer program stored on the memory.
According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the database drifting method of the present invention.
The positive progress effects of the invention are as follows:
the load of each database instance in the production environment is monitored, the database drifting is automatically carried out on the instance with unbalanced load according to the load condition, when the instance load is higher, the data of the corresponding database is drifted to the newly added instance, when the instance load is lower, the data of other databases in the production environment are drifted to the instance, the operation process is simple, manual guard is not needed, and meanwhile, in the drifting process, the whole process is ensured to be implemented in a controllable and safe range by checking the consistency of the data. By the method and the device, the load of the production environment can be dynamically balanced, the utilization rate of resources is improved, the interruption time of the service is shortened, the manual workload is reduced, and the production environment faults caused by manual operation errors are reduced.
Drawings
Fig. 1 is a schematic flow chart of a database drifting method according to embodiment 1 of the present invention.
Fig. 2 is a schematic flowchart of a database drifting method according to embodiment 2 of the present invention.
Fig. 3 is a schematic structural diagram of a database drifting system according to embodiment 3 of the present invention.
Fig. 4 is a schematic structural diagram of an electronic device according to embodiment 4 of the present invention.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Example 1
The present embodiment provides a database drifting method, as shown in fig. 1, the database drifting method includes the following steps:
In this embodiment, the first instance may be any instance in a production environment, and the first database refers to a database loaded in the first instance
In this embodiment, whether the load of the instance loading the database is too high (that is, the preset high load condition is met) may be determined by monitoring whether each item of performance index data corresponding to the database exceeds a preset upper limit load threshold, specifically, if the load exceeds the preset upper limit load threshold, the load is too high, and otherwise, the load is not too high.
In an alternative embodiment, the database index data includes, but is not limited to: the utilization rate of a CPU (central processing unit), wherein the larger the value of the utilization rate of the CPU is, the higher the load of the database instance is; QPS (requests per second), the larger the value of QPS, the higher the database instance load; the greater the DISK usage rate, the greater the value of the DISK usage rate, the higher the database instance load, for example, the CPU utilization rate may be greater than 70%, or the DISK usage rate exceeds 85% and cannot be cleared up is used as the upper limit load threshold.
And 102, when the first instance meets the high load condition, determining the data to be drifted in the first database, and creating a second database and a second instance.
When the load of the example to be detected meets a high load condition, entering a drifting process, wherein the database loaded by the example is a high load database, acquiring a part of data needing drifting from the high load database as data to be drifted, and the data to be drifted comprises but is not limited to service data and log files. A new database (i.e., the second database) and a new empty instance (i.e., the second instance) are created based on the size of the data to be drifted.
And 103, synchronizing the data to be drifted into the second database, and loading the second database to the second instance.
Specifically, a data synchronization link (defined herein as a first data synchronization link) is first created between the high-load database (i.e., the first database) and the new database (i.e., the second database) so that the data to be shifted in the high-load database is synchronized to the new database through the data synchronization link. As an alternative embodiment, the data synchronization link is generated between the high-load database and the new database based on the database identifier association, although the embodiment is not limited to the database identifier. Data synchronization is performed through the data synchronization link, and data drift is achieved.
In this embodiment, a part of data that needs to be shifted in the high-load database, that is, data to be shifted, is synchronized into the new database in a way of first full amount and then increment. As an optional implementation manner, when the data synchronization operation is performed for the first time, all the data to be drifted are synchronized into the new database, and the data to be drifted which is updated subsequently is synchronized into the new database in an incremental manner, so that the data to be drifted is synchronized into the new database in real time.
As an alternative, the drifting task may be automatically executed or automatically stopped according to a predefined time, for example, automatically executed during the low peak period of the service, although the embodiment is not limited to executing the drifting task during the low peak period of the service.
In the process of executing the drifting task, environment variables of the high-load database and the new database are checked, after data to be drifted are synchronized to the new database in real time each time, whether the data of the new database is consistent with a part of data (namely the data to be drifted) needing drifting in the high-load database is checked, and a subsequent flow switching task is executed after the data on the two sides are consistent.
And if the data of the new database is verified to be inconsistent with a part of data needing to be drifted in the high-load database, the flow switching is not carried out, and the part of data needing to be drifted in the high-load database is synchronized into the new database again. As an optional implementation manner, synchronizing a part of data needing to drift in the high-load database into the emptied new database in a full-volume manner; as an alternative, another embodiment, a part of the data in the high-load database that needs to be shifted is synchronized into the new database in an incremental manner, where the part of the data is different from the data in the new database.
And if the data of the new database is verified to be inconsistent with a part of data needing to be drifted in the high-load database after the data is resynchronized, judging that the drift task has errors. As an alternative embodiment, the drifting task is actively suspended, and the reason for the error of the drifting task is manually checked and removed, where the fault includes, but is not limited to, a database downtime or a network congestion. As an optional implementation manner, when the drift task is suspended, the timestamp of the last piece of data is reserved, that is, the data during the suspension period is stored, after the fault is eliminated, the stored data is synchronized to the new database, and then the latest data is synchronized, so that the time sequence of data synchronization is ensured.
As an optional implementation manner, if a failure that cannot be eliminated occurs, the whole drifting task may also be rolled back to ensure that the whole task is implemented within a controllable and safe range.
As an optional implementation manner, the task is set to be stopped at a predefined time, and after the predefined time is reached, the execution of the drifting task is considered to be completed; alternatively, when the performance index of the new database is just at the upper limit of the load threshold, the task is automatically stopped, and the execution of the drifting task is considered to be completed. And when the execution of the drifting task is finished, loading the new database to a new empty instance, and entering a flow switching process, wherein the new empty instance is used as an optional implementation mode for verifying whether the data to be drifted is consistent with the data of the new database, and judging whether the flow switching prerequisite is met. As an optional implementation manner, a DNS (domain name system) is used to switch traffic, and if the data to be drifted is consistent with the data in the new database, the traffic accessing the high-load database is forwarded to the new database through a data synchronization link; and if the data are inconsistent, re-entering the drifting task flow and executing the automatic troubleshooting processing.
After the traffic flow accessing the high-load database is switched to the new database, as an optional implementation manner, a test is performed at the application end, and if the test fails, the traffic flow accessing the new database is switched back to the high-load database. As an optional implementation manner, a prompt message is output to notify the relevant staff to perform manual troubleshooting processing.
And if the application end passes the test, disconnecting the data synchronization link between the high-load database and the new database. As an optional implementation manner, after the test passes, the space of the to-be-detected instance (i.e., the first instance) is cleaned, that is, the to-be-drifted data synchronized to the second database is deleted.
In the embodiment, the load of each database instance in the production environment is monitored, the data of the database is drifted to the newly added instance for the instance with higher load, the operation process is simple, manual guard is not needed, and meanwhile, in the drift process, the whole process is ensured to be implemented in a controllable and safe range by checking the consistency of the data. Through this embodiment, can realize dynamic balance in the load that satisfies the production environment, when the resource usage rate obtains promoting, shorten the interrupt time of business, reduced artificial work load to production environment trouble because of manual operation error brings has been reduced.
Example 2
The present embodiment provides another database drifting method, as shown in fig. 2, the database drifting method includes the following steps:
In this embodiment, the first instance may also be any instance in the production environment, and the first database also refers to the database loaded in the first instance.
In this embodiment, whether the load of the instance loading the database is too low (that is, a preset low load condition is met) may be determined by monitoring whether each item of performance index data corresponding to the database is lower than a preset lower load threshold, specifically, if the load exceeds the preset lower load threshold, it indicates that the load is too low, and otherwise, it indicates that the load is not too low.
As an alternative embodiment, the database metrics include, but are not limited to: the smaller the value of the CPU utilization rate is, the lower the load of the database instance is; QPS, the smaller the value of QPS is, the lower the load of the database instance is; the smaller the DISK usage rate, the lower the database instance load, for example, the DISK usage rate may be lower than 20% as a lower limit load threshold, and certainly, the upper limit load threshold is not specifically limited in this embodiment, and may be set as needed.
And when the load of the example to be detected meets the low-load condition, entering a drifting process, and setting the database loaded by the example as a low-load database. As an optional implementation manner, the loads of other instances are monitored, if an instance with a lower load is detected, the database loaded by the instance with the lower load is set as the database to be merged, and all the data is acquired from the database to be merged as the data to be drifted.
And step 203, synchronizing the data in the third database loaded by the third instance to the first database.
Specifically, a data synchronization link (defined herein as a second data synchronization link) is first created between the low-load database (i.e., the first database) and the database to be merged (i.e., the third database) so that the data in the database to be merged is synchronized to the low-load database through the data synchronization link. As an optional implementation manner, a data synchronization link is generated between the database to be merged and the low-load database based on the database identifier association, although this embodiment is not limited to be based on the database identifier. Data synchronization is performed through the data synchronization link, and data drift is achieved.
In this embodiment, the database to be merged may be one or more databases, and the data of the database to be merged, that is, the data to be drifted, are synchronized into the low-load database in a manner of first full amount and then increment. As an optional implementation manner, when the data synchronization operation is performed for the first time, all the data to be drifted are synchronized into the low-load database, and then the data to be drifted, which is subsequently updated by the database to be merged, is synchronized into the low-load database in an incremental manner, so that the data to be drifted is synchronized into the low-load database in real time.
In this embodiment, after the data to be drifted is synchronized to the low-load database, each item of performance index data corresponding to the low-load database is monitored, and it is ensured that each item of performance index data is not higher than the upper limit load threshold.
As an alternative, the drifting task may be automatically executed or automatically stopped according to a predefined time, for example, automatically executed during the low peak period of the service, although the embodiment is not limited to executing the drifting task during the low peak period of the service.
In the process of executing the drifting task, checking the environment variables of the low-load database and the database to be merged, after synchronizing the data to be drifted to the low-load database in real time each time, checking whether the data of the low-load database is consistent with the data (namely the data to be drifted) in the database to be merged, and executing a subsequent flow switching task after ensuring that the data on the two sides are consistent.
And if the data of the database to be merged is not consistent with the newly added data synchronized in the low-load database, the flow switching is not carried out, and the data of the database to be merged is synchronized into the low-load database again. As an optional implementation manner, the synchronized low-load database is restored to the low-load database before synchronization, and the data of the databases to be merged are synchronized to the low-load database before synchronization in a full-scale manner; as another optional embodiment, the data of the database to be merged and the different parts of the synchronized new data in the low-load database are synchronized into the low-load database in an incremental manner.
And if the synchronized partial data in the low-load database is verified to be still inconsistent with the data of the database to be merged after the data is resynchronized, judging that the drift task has errors. As an alternative embodiment, the drifting task is actively suspended, and the reason for the error of the drifting task is manually checked and removed, where the fault includes, but is not limited to, a database downtime or a network congestion. As an optional implementation manner, when the drift task is suspended, the timestamp of the last piece of data is reserved, that is, the data during the suspension period is stored, and after the stored data is synchronized to the low-load database after the fault is eliminated, the latest data is synchronized, so that the time sequence of data synchronization is ensured.
As an optional implementation manner, if a failure that cannot be eliminated occurs, the whole drifting task may also be rolled back to ensure that the whole task is implemented within a controllable and safe range.
As an optional implementation manner, setting the predefined time as stopping the task, and after the predefined time is reached, considering that the execution of the drifting task is completed; alternatively, when the performance index of the low-load database is just at the upper limit of the load threshold, the task is automatically stopped, and the execution of the drifting task is considered to be completed. After the execution of the drifting task is completed, a flow switching process is entered, as an optional implementation manner, whether the data to be drifted is consistent with the newly added data synchronized in the low-load database is checked, and whether a flow switching prerequisite condition is met is judged. As an optional implementation manner, a DNS is used to switch traffic, and if data to be drifted is consistent with newly added data after synchronization in the low-load database, service traffic accessing the database to be merged is forwarded to the low-load database through a data synchronization link; and if the data are inconsistent, re-entering the drifting task flow and executing the automatic troubleshooting processing.
After the service traffic accessing the database to be merged is forwarded to the low-load database after the drift task, as an optional implementation manner, the test is performed at the application end, and if the test fails, the service traffic is switched back to the database to be merged again. As an optional implementation manner, a prompt message is output to notify the relevant staff to perform manual troubleshooting processing.
And if the application end passes the test, disconnecting the data synchronization link between the data of the database to be merged and the low-load database. As an optional embodiment, after the test passes, the space in which the instance of the database to be merged (i.e., the third instance) is loaded is cleaned, i.e., the data to be migrated that has been synchronized to the first database is deleted. As an optional implementation manner, if all the data in the database to be merged are deleted, the entire database to be merged is deleted.
In the embodiment, the load of each database instance in the production environment is monitored, the data of other databases in the production environment are drifted to the instance for the instance with lower load, the operation process is simple, manual guard is not needed, and meanwhile, in the drift process, the whole process is ensured to be implemented in a controllable and safe range by checking the consistency of the data. Through this embodiment, can realize dynamic balance in the load that satisfies the production environment, when the resource usage rate obtains promoting, shorten the interrupt time of business, reduced artificial work load to production environment trouble because of manual operation error brings has been reduced.
Example 3
As shown in fig. 3, the database drifting system includes a load detection module 30, a first drifting preparation module 31, a first data synchronization module 32, a first link building module 33, a first data verification module 34, a first traffic switching module 35, a second drifting preparation module 36, a second data synchronization module 37, a second link building module 38, a second data verification module 39, and a second traffic switching module 310.
The load detection module 30 is configured to monitor whether each item of performance index data corresponding to the database exceeds a preset upper limit load threshold, to determine whether an instance loading the database is overloaded (that is, meets a preset high load condition), specifically, if the load exceeds the preset upper limit load threshold, the load is overloaded, and otherwise, the load is not overloaded.
In an alternative embodiment, the database index data includes, but is not limited to: the utilization rate of a CPU (central processing unit), wherein the larger the value of the utilization rate of the CPU is, the higher the load of the database instance is; QPS (requests per second), the larger the value of QPS, the higher the database instance load; the greater the DISK usage rate, the greater the value of the DISK usage rate, the higher the database instance load, for example, the load detection module 30 may use the CPU utilization rate greater than 70% or the DISK usage rate exceeding 85% and being unable to be cleaned as the upper limit load threshold, which is not specifically limited in this embodiment, and may be set as needed.
The first drift preparation module 31 is configured to enter a drift process when the load of the to-be-detected instance meets a high load condition, where the database loaded in the instance is a high load database, and a part of data that needs to drift is acquired from the high load database as data to be drifted, where the data to be drifted includes, but is not limited to, service data and log files. A new database (i.e., second data) and a new empty instance (second instance) are created based on the size of the data to be drifted.
Specifically, first, the first link building module 33 creates a data synchronization link (defined as a first data synchronization link herein) between the high-load database (i.e., the first database) and the new database (i.e., the second database), so that the first data synchronization module 32 synchronizes the data to be drifted in the high-load database to the new database through the data synchronization link. As an optional embodiment, the first link building module 33 generates a data synchronization link between the high-load database and the new database based on the database identifier association, but of course, this embodiment is not limited to the database identifier. Data synchronization is performed over the data synchronization link, and data drift is achieved.
In this embodiment, the first data synchronization module 32 synchronizes a part of data that needs to be shifted in the high-load database, that is, data to be shifted, to the new database in a manner of first full amount and then increment. As an optional implementation manner, when the data synchronization operation is performed for the first time, the first data synchronization module 32 synchronizes all the data to be drifted into the new database, and synchronizes the subsequently updated data to be drifted into the new database in an incremental manner, so as to implement real-time synchronization of the data to be drifted into the new database.
As an alternative, the first data synchronization module 32 may automatically perform or automatically stop the drifting task according to a predefined time, for example, automatically perform the task at a low peak of the service, although the embodiment is not limited to performing the drifting task at the low peak of the service.
In the process of executing the drifting task, the first data checking module 34 checks the environment variables of the high-load database and the new database, and after synchronizing the data to be drifted to the new database in real time each time, checks whether the data of the new database is consistent with a part of data (i.e., the data to be drifted) to be drifted in the high-load database, and executes the subsequent flow switching task after ensuring that the data on both sides are consistent.
If the first data checking module 34 checks that the data of the new database is inconsistent with a part of data needing to be shifted in the high-load database, the flow switching is not performed, and the part of data needing to be shifted in the high-load database is synchronized into the new database again. As an optional implementation manner, the first data synchronization module 32 synchronizes a part of data that needs to be shifted in the high-load database to the emptied new database in a full-scale manner; as an alternative, the first data synchronization module 32 synchronizes a part of the data of the high-load database that needs to be shifted and a part of the data of the new database that is different from the data of the high-load database into the new database in an incremental manner.
If the data in the new database is not consistent with a part of data needing to be shifted in the high-load database after the data is resynchronized, the first data checking module 34 determines that the shift task has an error. As an alternative embodiment, the drifting task is actively suspended, and the reason for the error of the drifting task is manually checked and removed, where the fault includes, but is not limited to, a database downtime or a network congestion. As an optional implementation manner, when the drift task is suspended, the timestamp of the last piece of data is reserved, that is, the data during the suspension period is stored, after the fault is eliminated, the stored data is synchronized to the new database, and then the latest data is synchronized, so that the time sequence of data synchronization is ensured.
As an optional implementation manner, if a failure that cannot be eliminated occurs, the whole drifting task may also be rolled back to ensure that the whole task is implemented within a controllable and safe range, and as an optional implementation manner, after rolling back, the first data checking module 34 checks the data of the new database and a part of data that needs to be shifted in the high-load database, so as to ensure data synchronization on both sides.
As an optional implementation manner, the first data synchronization module 32 sets to stop the task at a predefined time, and after the predefined time is reached, it is regarded that the execution of the drifting task is completed; alternatively, when the performance index of the new database is just at the upper limit of the load threshold, the task is automatically stopped, and the execution of the drifting task is considered to be completed.
After the execution of the drift task is completed, the new database is loaded to the new empty instance, and the flow switching process is performed, as an optional implementation manner, the first data checking module 34 checks whether the data to be drifted is consistent with the data of the new database, and determines whether the flow switching prerequisite is met. As an optional implementation manner, a DNS (domain name system) is used to switch traffic, and if the data to be drifted is consistent with the data in the new database, the first traffic switching module 35 forwards the traffic accessing the high-load database to the new database through the data synchronization link; and if the data are inconsistent, re-entering the drifting task flow and executing the automatic troubleshooting processing.
After switching the traffic flow accessing the high load database to the new database, as an optional implementation manner, a test is performed at the application end, and if the test fails, the first traffic switching module 35 switches the traffic flow accessing the new database back to the high load database. As an optional implementation manner, a prompt message is output to notify the relevant staff to perform manual troubleshooting processing.
If the application passes the test, the first link establishment module 33 disconnects the data synchronization link between the high-load database and the new database. As an optional embodiment, after the test is passed, the first data synchronization module 32 cleans up the space of the to-be-detected instance (i.e., the first instance), that is, deletes the to-be-drifted data that has been synchronized to the second database.
The load detection module 30 is further configured to monitor whether each performance index data corresponding to the database is lower than a preset lower load threshold, to determine whether the load of the instance loading the database is too low (that is, the preset low load condition is met), specifically, if the load exceeds the preset lower load threshold, the load is too low, and otherwise, the load is not too low.
As an alternative embodiment, the database metrics include, but are not limited to: the smaller the value of the CPU utilization rate is, the lower the load of the database instance is; QPS, the smaller the value of QPS is, the lower the load of the database instance is; the smaller the DISK usage rate, the lower the value of the DISK usage rate, the lower the load of the database instance, for example, the load detection module 30 may use the DISK usage rate lower than 20% as the lower limit load threshold, which is, of course, not specifically limited by the embodiment, and may be set according to the need.
The second drift preparation module 36 is configured to enter a drift process when the load of the to-be-detected instance meets the low-load condition, and set the database loaded by the instance as a low-load database. As an optional implementation manner, the loads of other instances are monitored, if an instance with a lower load is detected, the second drift preparation module 36 sets the database loaded by the instance with the lower load as the database to be merged, and obtains all the data from the database to be merged as the data to be drifted.
Specifically, first, the second link building module 38 creates a data synchronization link (defined herein as a second data synchronization link) between the low-load database (i.e., the first database) and the database to be merged (i.e., the third database), so that the second data synchronization module 37 synchronizes the data in the database to be merged to the low-load database through the data synchronization link. As an optional embodiment, the second link building module 38 generates a data synchronization link between the database to be merged and the low-load database based on the database identifier association, but this embodiment is not limited to be based on the database identifier. Data synchronization is performed through the data synchronization link, and data drift is achieved.
In this embodiment, the database to be merged may be one or more databases, and the second data synchronization module 37 synchronizes the data of the database to be merged, that is, the data to be drifted, to the low-load database in a full-first and then incremental manner. As an optional implementation manner, when the data synchronization operation is performed for the first time, the second data synchronization module 37 synchronizes all the data to be drifted into the low-load database, and synchronizes the data to be drifted, which is updated subsequently by the database to be merged, into the low-load database in an incremental manner, so as to implement real-time synchronization of the data to be drifted into the low-load database.
In this embodiment, after the data to be drifted is synchronized to the low-load database, the load detection module 30 monitors each item of performance index data corresponding to the low-load database, and ensures that each item of performance index data is not higher than the upper limit load threshold.
As an alternative, the second data synchronization module 37 may automatically perform or automatically stop the drifting task according to a predefined time, for example, automatically perform the task at a low peak of the service, although the embodiment is not limited to performing the drifting task at the low peak of the service.
In the process of executing the drifting task, the second data checking module 39 checks the environment variables of the low-load database and the database to be merged, and after synchronizing the data to be drifted to the low-load database in real time each time, checks whether the data of the low-load database is consistent with the data (i.e., the data to be drifted) in the database to be merged, and executes the subsequent flow switching task after ensuring that the data on both sides are consistent.
If the second data checking module 39 checks that the data of the database to be merged is not consistent with the new data synchronized in the low-load database, the flow switching is not performed, and the data of the database to be merged is synchronized into the low-load database again. As an optional implementation manner, the second data synchronization module 37 restores the synchronized low-load database to the low-load database before synchronization, and synchronizes the data of the database to be merged to the low-load database before synchronization in a full-scale manner; as an alternative embodiment, the second data synchronization module 37 synchronizes the data of the database to be merged and the different part of the synchronized new data in the low-load database to the low-load database in an incremental manner.
If the part of the synchronized data in the low-load database is still inconsistent with the data in the database to be merged after the data is resynchronized, the second data checking module 39 determines that the drift task has an error. As an alternative embodiment, the drifting task is actively suspended, and the reason for the error of the drifting task is manually checked and removed, where the fault includes, but is not limited to, a database downtime or a network congestion. As an optional implementation manner, when the drift task is suspended, the timestamp of the last piece of data is reserved, that is, the data during the suspension period is stored, and after the stored data is synchronized to the low-load database after the fault is eliminated, the latest data is synchronized, so that the time sequence of data synchronization is ensured.
As an optional implementation manner, if a fault which cannot be eliminated occurs, the whole drifting task may also be rolled back to ensure that the whole task is implemented within a controllable and safe range, and as an optional implementation manner, after rolling back, the second data inspection module 39 checks partial data synchronized in the low-load database and data of the database to be merged to ensure data synchronization on both sides.
As an optional implementation manner, the second data synchronization module 37 sets that the task is stopped at a predefined time, and after the predefined time is reached, it is considered that the execution of the drifting task is completed; alternatively, when the performance index of the low-load database is just at the upper limit of the load threshold, the task is automatically stopped, and the execution of the drifting task is considered to be completed.
As an optional implementation, the second data checking module 39 checks whether the data to be drifted is consistent with the new data synchronized in the low-load database, and determines whether the traffic switching prerequisite is satisfied. As an optional implementation manner, a DNS is used to switch traffic, and if the data to be drifted is consistent with the newly added synchronized data in the low-load database, the second traffic switching module 310 forwards the traffic accessing the database to be merged to the low-load database through the data synchronization link; and if the data are inconsistent, re-entering the drifting task flow and executing the automatic troubleshooting processing.
After the service traffic accessing the database to be merged is forwarded to the low-load database after the drift task, as an optional implementation manner, a test is performed at the application end, and if the test fails, the second traffic switching module 310 switches the service traffic back to the database to be merged again. As an optional implementation manner, a prompt message is output to notify the relevant staff to perform manual troubleshooting processing.
If the application end passes the test, the second link establishment module 38 disconnects the data synchronization link between the data of the database to be merged and the low-load database. As an optional embodiment, after the test passes, the second data synchronization module 37 cleans up the space where the instances of the databases to be merged (i.e. the third instances) are loaded, i.e. deletes the data to be migrated that has been synchronized to the first database. As an optional implementation manner, if all the data in the database to be merged are deleted, the entire database to be merged is deleted.
According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor connected to the memory, the processor implementing the database drifting method of the present invention when executing a computer program stored on the memory.
In the embodiment, the load of each database instance in the production environment is monitored, for the instance with unbalanced load, the database drifting is automatically performed according to the condition of the load, when the instance load is higher, the data of the database is drifted into a newly added instance, when the instance load is lower, the data of other databases in the production environment are drifted into the instance, the operation process is simple, manual guard is not needed, and meanwhile, in the drifting process, the whole process is ensured to be implemented in a controllable and safe range by checking the consistency of the data. Through this embodiment, can realize dynamic balance in the load that satisfies the production environment, when the resource usage rate obtains promoting, shorten the interrupt time of business, reduced artificial work load to production environment trouble because of manual operation error brings has been reduced.
Example 4
Fig. 4 is a schematic structural diagram of an electronic device provided in this embodiment. The electronic equipment comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and the processor executes the program to realize the e-commerce platform user information configuration method of the embodiment 1. The electronic device 40 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in FIG. 4, electronic device 40 may take the form of a general purpose computing device, which may be a server device, for example. The components of electronic device 40 may include, but are not limited to: the at least one processor 41, the at least one memory 42, and a bus 43 connecting the various system components (including the memory 42 and the processor 41).
The bus 43 includes a data bus, an address bus, and a control bus.
The memory 42 may include volatile memory, such as Random Access Memory (RAM)421 and/or cache memory 422, and may further include Read Only Memory (ROM) 423.
The processor 41 executes various functional applications and data processing, such as the database drifting method of embodiment 1 and embodiment 2 of the present invention, by running the computer program stored in the memory 42.
The electronic device 40 may also communicate with one or more external devices 44 (e.g., keyboard, pointing device, etc.). Such communication may be through an input/output (I/O) interface 45. Also, model-generating device 40 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via network adapter 46. As shown, the network adapter 46 communicates with the other modules of the model-generated device 40 over a bus 43. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the model-generating device 40, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
Example 5
The present embodiment provides a computer-readable storage medium on which a computer program is stored, which when executed by a processor, implements the steps of the database drifting methods of embodiments 1 and 2.
More specific examples, among others, that the readable storage medium may employ may include, but are not limited to: a portable disk, a hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible implementation, the invention can also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps of implementing the database drifting methods of embodiments 1 and 2, when said program product is run on said terminal device.
Where program code for carrying out the invention is written in any combination of one or more programming languages, the program code may be executed entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device or entirely on the remote device.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (14)
1. A database drifting method, comprising the steps of:
detecting whether a first instance loaded with a first database meets a preset high-load condition;
when the first instance meets the high load condition, determining data to be drifted in the first database, and creating a second database and a second instance;
synchronizing the data to be drifted into the second database and loading the second database to the second instance.
2. The database drifting method according to claim 1, wherein between the step of creating the second database and the second instance and the step of synchronizing the data to be drifted into the second database, further comprising:
establishing a first data synchronization link between the second database and the first database so as to synchronize the data to be drifted into the second database through the first data synchronization link.
3. The database drifting method of claim 1, further comprising, after the step of synchronizing the data to be drifted into the second database:
checking whether the data synchronized to the second database is consistent with the data to be drifted;
and if the verification is passed, switching the service flow accessing the first database to the second database.
4. The database drifting method according to claim 1, further comprising:
detecting whether the first instance meets a preset low load condition;
when the first instance meets the low load condition, determining a third instance to be merged from instances other than the first instance, and synchronizing data in a third database loaded by the third instance into the first database, wherein the first instance after synchronization does not meet the high load condition.
5. The database drifting method according to claim 4, wherein between the step of determining a third instance to be merged from the instances other than the first instance and the step of synchronizing data in a third database loaded by the third instance into the first database, further comprising:
establishing a second data synchronization link between the third database and the first database so as to synchronize data in the third database to the first database through the second data synchronization link.
6. The database drifting method according to claim 4, further comprising, after the step of synchronizing the data in the database loaded by the third instance into the first database:
checking whether the data synchronized to the first database is consistent with the data in the third database;
and if the verification is passed, switching the service flow accessing the third database to the first database.
7. The database drifting system is characterized by comprising a load detection module, a first drifting preparation module and a first data synchronization module;
the load detection module is used for detecting whether a first instance loaded with a first database meets a preset high load condition;
the first drift preparation module is used for determining data to be drifted in the first database and creating a second database and a second example when the first example meets the high load condition;
the first data synchronization module is used for synchronizing the data to be drifted into the second database and loading the second database to the second instance.
8. The database drift system of claim 7, further comprising a first link construction module;
the first link building module is used for building a first data synchronization link between the second database and the first database so as to synchronize the data to be drifted into the second database through the first data synchronization link.
9. The database drift system of claim 7, further comprising a first data verification module and a first traffic switching module;
the first data checking module is used for checking whether the data synchronized to the second database is consistent with the data to be drifted;
and the first traffic switching module is used for switching the service traffic accessing the first database to the second database when the check is passed.
10. The database drift system of claim 7, wherein the load detection module is further configured to detect whether the first instance meets a preset low load condition;
the database drift module further comprises:
a second drift preparation module for determining a third instance to be merged from instances other than the first instance when the first instance satisfies the low load condition;
a second data synchronization module, configured to synchronize data in a third database loaded by the third instance into the first database, where the first instance after synchronization does not satisfy the high load condition.
11. The database drift system of claim 10, further comprising a second link construction module;
the second link building module is used for building a second data synchronization link between the third database and the first database so as to synchronize the data in the third database to the first database through the second data synchronization link.
12. The database drifting system of claim 10, further comprising a second data verification module and a second traffic switching module:
the second data checking module is used for checking whether the data synchronized to the first database is consistent with the data in the third database;
and the second traffic switching module is used for switching the service traffic accessing the third database to the first database when the check is passed.
13. An electronic device, comprising a memory and a processor connected to the memory, the processor implementing the database drifting method of any one of the preceding claims 1-6 when executing a computer program stored on the memory.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a database drifting method according to any one of the preceding claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111143652.7A CN113987065A (en) | 2021-09-28 | 2021-09-28 | Database drifting method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111143652.7A CN113987065A (en) | 2021-09-28 | 2021-09-28 | Database drifting method, system, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113987065A true CN113987065A (en) | 2022-01-28 |
Family
ID=79737049
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111143652.7A Pending CN113987065A (en) | 2021-09-28 | 2021-09-28 | Database drifting method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113987065A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024087914A1 (en) * | 2022-10-24 | 2024-05-02 | 超聚变数字技术有限公司 | Data synchronization method and computing device |
-
2021
- 2021-09-28 CN CN202111143652.7A patent/CN113987065A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024087914A1 (en) * | 2022-10-24 | 2024-05-02 | 超聚变数字技术有限公司 | Data synchronization method and computing device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106951559B (en) | Data recovery method in distributed file system and electronic equipment | |
| US7873732B2 (en) | Maintaining service reliability in a data center using a service level objective provisioning mechanism | |
| US20050268153A1 (en) | Method of solving a split-brain condition | |
| CN111953566B (en) | Distributed fault monitoring-based method and virtual machine high-availability system | |
| US10924538B2 (en) | Systems and methods of monitoring software application processes | |
| CN110457176B (en) | Monitoring method and device for distributed system, storage medium and electronic equipment | |
| CN103595572B (en) | A kind of method of cloud computing cluster interior joint selfreparing | |
| CN109144789A (en) | A kind of method, apparatus and system for restarting OSD | |
| CN102902615A (en) | Failure alarm method and system for Lustre parallel file system | |
| CN110063042A (en) | A kind of response method and its terminal of database failure | |
| CN113987065A (en) | Database drifting method, system, electronic device and storage medium | |
| US20110225463A1 (en) | Detecting and recovering from process failures | |
| CN115632706B (en) | FC link management method, device, equipment and readable storage medium | |
| CN111104266A (en) | Access resource allocation method and device, storage medium and electronic equipment | |
| CN106411643A (en) | BMC (Baseboard Management Controller) detection method and device | |
| CN114356533B (en) | Micro-service non-perception issuing system and method, electronic equipment and storage medium | |
| CN115686831A (en) | Task processing method and device based on distributed system, equipment and medium | |
| CN115686368A (en) | Method, system, apparatus and medium for storage capacity expansion of nodes of block chain network | |
| CN112269693B (en) | Node self-coordination method, device and computer readable storage medium | |
| CN105511952A (en) | Resource self-migration method and system based on cloud computing platform | |
| WO2022009438A1 (en) | Server maintenance control device, system, control method, and program | |
| CN114039848A (en) | Method, device and equipment for realizing high availability of InCloudInsight management platform | |
| CN110287066B (en) | Server partition migration method and related device | |
| CN117971564B (en) | Data recovery method, device, computer equipment and storage medium | |
| CN117762740A (en) | Method, system, equipment and medium for data security monitoring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |