CN116467386A - Method and device for concurrent playback of backup database data - Google Patents
Method and device for concurrent playback of backup database data Download PDFInfo
- Publication number
- CN116467386A CN116467386A CN202310391715.3A CN202310391715A CN116467386A CN 116467386 A CN116467386 A CN 116467386A CN 202310391715 A CN202310391715 A CN 202310391715A CN 116467386 A CN116467386 A CN 116467386A
- Authority
- CN
- China
- Prior art keywords
- transaction
- playback
- data
- buf
- transactions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 239000012634 fragment Substances 0.000 claims description 6
- 238000012163 sequencing technique Methods 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5011—Pool
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of database data processing, in particular to a method and a device for concurrent playback of backup database data, which comprises the following steps: s1, when the standby library receives a transaction event, the transaction data are added into the BUF. S2, adding the transaction for executing concurrent playback into a thread pool TXNLIST; s3, the number of the workers in the thread pool is configurable according to the number of the CPU and the load condition of the processor. Compared with the prior art, the method and the device have the advantages that under the scene of main and standby copying, a thread pool is introduced into the standby library, and after a CHECKPOINT event is received, concurrent playback is performed on the transaction and the data in the BUF, so that the data copying efficiency is improved, and the consistency of the main and standby data is ensured.
Description
Technical Field
The invention relates to the technical field of database data processing, and particularly provides a method and a device for concurrent playback of backup database data.
Background
When the data of the main library is copied, and the data of the backup library is played back in huge data volume under the condition of the concurrency of the main library and the large data copying volume, the data of the backup library is seriously accumulated and the efficiency of the copying function is seriously influenced when the single concurrency playback speed is not enough to catch up with the data of the main library.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a method for concurrently playing back backup database data with strong practicability.
The invention further aims to provide a safe and applicable backup database data concurrent playback device which is reasonable in design.
The technical scheme adopted for solving the technical problems is as follows:
a method for concurrent playback of backup data comprises the following steps:
s1, when the standby library receives a transaction event, the transaction data are added into the BUF.
S2, adding the transaction for executing concurrent playback into a thread pool TXNLIST;
s3, the number of the workers in the thread pool is configurable according to the number of the CPU and the load condition of the processor.
Further, in step S1, when the standby accepts the transaction event, the transaction data is added to the BUF and ordered according to the commit timestamp of each transaction.
Further, in step S1, after the backup repository receives the chekpoint event, the parallel playback transactions in the BUF are queried, that is, the execution of the transactions is not dependent on each other, and the distributed data is the data fragments with different transaction operations, so that the parallel playback transactions are played back concurrently.
Further, in step S2, the idle worker may perform playback and landing operations on the TXN added to the queue, and write the corresponding transaction data into the storage.
Further, in step S3, after the playback of the transaction is completed, the own transaction information is deleted from the BUF, the transaction played back in the BUF is queried again, and if found, the transaction is added again to the playback thread pool for concurrent playback.
Further, if no playback transaction is found, wait for the next check point event and process the transaction playback task again.
When the backup database receives a transaction event, the transaction data is added into the BUF, the transaction for executing the concurrent playback is added into a thread pool TXLIST, and the number of workers in the thread pool is configurable according to the number of CPU (Central processing Unit) and the load condition of the processor.
Further, when the backup library receives the transaction event, adding the transaction data into the BUF, and sequencing according to the commit timestamp of each transaction;
after the backup library receives the CHECKPOINT event, the parallel playback transactions in the BUF are inquired, namely the execution of the transactions is not interdependent, and the parallel playback transactions are played back in parallel when the distributed data are different data fragments operated by different transactions.
Further, the idle worker can play back the disk-dropping operation on the TXN added into the queue, and write the corresponding transaction data into the storage.
Further, the number of the workers in the thread pool is configurable according to the number of the CPU and the load condition of the processor, after the transaction playback is finished, own transaction information is deleted in the BUF, the transactions which can be played back in the BUF are queried again, and if the transactions are found, the transactions are added into the playback thread pool again for concurrent playback;
if no playback transaction is found, waiting for the next check point event and processing the transaction playback task again.
Compared with the prior art, the method and the device for concurrent playback of the backup database data have the following outstanding beneficial effects:
the invention introduces a thread pool in the backup library under the scene of the primary and backup copying, and plays back the transaction and the data in the BUF concurrently after receiving the CHECKPOINT event, thereby improving the data copying efficiency and ensuring the primary and backup data consistency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for concurrent playback of backup data.
Detailed Description
In order to provide a better understanding of the aspects of the present invention, the present invention will be described in further detail with reference to specific embodiments. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
A preferred embodiment is given below:
as shown in fig. 1, a method for concurrent playback of backup data in this embodiment includes the following steps:
and (2) when the standby library receives the transaction event, adding the transaction data into the BUF, and sequencing according to the commit timestamp of each transaction.
And (2) and (3) after the backup library receives the CHECKPOINT event, inquiring the transactions which can be played back in parallel in the BUF, namely, the execution of the transactions is not interdependent, and different data fragments are operated for different transactions in the distributed data. The transactions which can be played back in parallel, so that the playback speed is improved, and the consistency of the main data and the standby data is not affected.
Step (4) adds the transaction that can perform concurrent playback to the TXNLIST in the thread pool.
And (5) the idle worker in the step (5) and the step (6) can perform playback and disk dropping operation on the corresponding transaction data for the TXN added into the queue, and write the corresponding transaction data into the storage.
The number of workers in the thread pool is configurable according to the number of processor CPUs and the load condition.
And (7) deleting own transaction information in the BUF after the transaction playback is finished.
And (8) after the transaction playback is finished, querying the transaction which can be played back in the BUF again, and if the transaction is found, adding the transaction into a playback thread pool again for concurrent playback.
And (9) if no transaction capable of being played back is found, waiting for the next check point event, and processing the transaction playback task again.
Based on the above method, in the device for concurrent playback of backup database data in this embodiment, when the backup database receives a transaction event, the transaction data is added to the BUF, the transaction for executing the concurrent playback is added to the TXNLIST in the thread pool, and the number of workers in the thread pool is configurable according to the number of CPUs and the load condition of the processor.
When the backup library receives the transaction event, adding the transaction data into the BUF, and sequencing according to the commit timestamp of each transaction;
after the backup library receives the CHECKPOINT event, the parallel playback transactions in the BUF are inquired, namely the execution of the transactions is not interdependent, and the parallel playback transactions are played back in parallel when the distributed data are different data fragments operated by different transactions.
And the idle worker can play back the disc-dropping operation on the TXN added into the queue and write the corresponding transaction data into the storage.
The number of the workers in the thread pool is configurable according to the number of the CPU and the load condition of the processor, after the transaction playback is finished, the own transaction information is deleted in the BUF, the transactions which can be played back in the BUF are queried again, and if the transactions are found, the transactions are added into the playback thread pool again for concurrent playback;
if no playback transaction is found, waiting for the next check point event and processing the transaction playback task again.
The above specific embodiments are merely specific examples of the present invention, and the scope of the present invention includes, but is not limited to, the specific embodiments described above, any suitable changes or substitutions made by one of ordinary skill in the art, which are consistent with the present invention, of a method and apparatus for concurrent playback of database data, and all fall within the scope of the present invention.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. A method for concurrently playing back backup data is characterized by comprising the following steps:
s1, when a standby library receives a transaction event, adding transaction data into BUF;
s2, adding the transaction for executing concurrent playback into a thread pool TXNLIST;
s3, the number of the workers in the thread pool is configurable according to the number of the CPU and the load condition of the processor.
2. A method of concurrent playback of database data according to claim 1, characterized in that in step S1, when the database accepts a transaction event, the transaction data is added to the BUF and ordered according to the commit timestamp of each transaction.
3. The method for concurrent playback of backup data according to claim 2, wherein in step S1, after the backup receives the chekpoint event, the backup queries parallel playback transactions in the BUF, that is, the execution of the transactions is not interdependent, and the parallel playback transactions are concurrently played back when the distributed data is different data fragments operated by different transactions.
4. The method for concurrent playback of backup data according to claim 3, wherein in step S2, the idle worker can perform playback landing operation on the corresponding transaction data for the TXN added to the queue, and write the corresponding transaction data into the storage.
5. The method for concurrent playback of database data according to claim 4, wherein in step S3, after the transaction playback is completed, the transaction information of the transaction is deleted from the BUF, the transaction played back in the BUF is queried again, and if found, the transaction is added again to the playback thread pool for concurrent playback.
6. The method for concurrent playback of database data as set forth in claim 5, wherein if no playback transaction is found, waiting for the next check point event and processing the transaction playback task again.
7. The backup database data concurrency playback device is characterized in that when a backup database receives a transaction event, the transaction data is added into BUF, the transaction for executing concurrency playback is added into a thread pool TXLIST, and the number of workers in the thread pool is configurable according to the number of CPU (Central processing units) and the load condition of a processor.
8. The backup data concurrency playback apparatus of claim 7, wherein when the backup accepts the transaction event, the transaction data is added to the BUF and ordered according to the commit timestamp of each transaction;
after the backup library receives the CHECKPOINT event, the parallel playback transactions in the BUF are inquired, namely the execution of the transactions is not interdependent, and the parallel playback transactions are played back in parallel when the distributed data are different data fragments operated by different transactions.
9. The device for concurrent playback of backup data as claimed in claim 8, wherein the idle worker is capable of performing a playback drop operation on the corresponding transaction data for the TXN added to the queue, and writing the corresponding transaction data to the storage.
10. The device for concurrent playback of database data according to claim 9, wherein the number of workers in the thread pool is configurable according to the number of CPUs and load conditions of the processors, after the transaction playback is finished, the own transaction information is deleted in the BUF, the transactions which can be played back in the BUF are queried again, and if found, the transactions are added into the playback thread pool again for concurrent playback;
if no playback transaction is found, waiting for the next check point event and processing the transaction playback task again.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310391715.3A CN116467386A (en) | 2023-04-13 | 2023-04-13 | Method and device for concurrent playback of backup database data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310391715.3A CN116467386A (en) | 2023-04-13 | 2023-04-13 | Method and device for concurrent playback of backup database data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116467386A true CN116467386A (en) | 2023-07-21 |
Family
ID=87172960
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310391715.3A Pending CN116467386A (en) | 2023-04-13 | 2023-04-13 | Method and device for concurrent playback of backup database data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116467386A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116932603A (en) * | 2023-08-03 | 2023-10-24 | 上海沄熹科技有限公司 | High-performance transaction concurrency copying method based on directed graph |
-
2023
- 2023-04-13 CN CN202310391715.3A patent/CN116467386A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116932603A (en) * | 2023-08-03 | 2023-10-24 | 上海沄熹科技有限公司 | High-performance transaction concurrency copying method based on directed graph |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11914572B2 (en) | Adaptive query routing in a replicated database environment | |
| US10180946B2 (en) | Consistent execution of partial queries in hybrid DBMS | |
| US6961865B1 (en) | Techniques for resuming a transaction after an error | |
| US7996363B2 (en) | Real-time apply mechanism in standby database environments | |
| US20120023369A1 (en) | Batching transactions to apply to a database | |
| US10366075B2 (en) | Database management system and method | |
| US20130117236A1 (en) | Database Log Replay Parallelization | |
| US9652491B2 (en) | Out-of-order execution of strictly-ordered transactional workloads | |
| US9542279B2 (en) | Shadow paging based log segment directory | |
| WO2019109854A1 (en) | Data processing method and device for distributed database, storage medium, and electronic device | |
| WO2022002103A1 (en) | Method for playing back log on data node, data node, and system | |
| Yang et al. | F1 Lightning: HTAP as a Service | |
| CN116467386A (en) | Method and device for concurrent playback of backup database data | |
| US11176004B2 (en) | Test continuous log replay | |
| CN114116665B (en) | Method for writing transaction log in parallel in database to promote processing efficiency | |
| US9965355B2 (en) | System and method for dynamic collection of system management data in a mainframe computing environment | |
| Yuan et al. | Rubato DB: A highly scalable staged grid database system for OLTP and big data applications | |
| WO2024098363A1 (en) | Multicore-processor-based concurrent transaction processing method and system | |
| CN117112522A (en) | Concurrent process log management method, device, equipment and storage medium | |
| US10387283B2 (en) | Dynamically adjusting statistics collection time in a database management system | |
| CN115904817A (en) | Distributed database parallel playback method and device, electronic equipment and storage medium | |
| Low et al. | Scalability of database bulk insertion with multi-threading | |
| US20170031597A1 (en) | Methods and systems of garbage collection and defragmentation in a distributed database | |
| WO2020140623A1 (en) | Electronic device, metadata processing method and computer readable storage medium | |
| CN109710690B (en) | Service driving calculation method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |