CN110347482B - OLTP transaction combination rule and queue model improvement method based on OLTPShare - Google Patents
OLTP transaction combination rule and queue model improvement method based on OLTPShare Download PDFInfo
- Publication number
- CN110347482B CN110347482B CN201910651419.6A CN201910651419A CN110347482B CN 110347482 B CN110347482 B CN 110347482B CN 201910651419 A CN201910651419 A CN 201910651419A CN 110347482 B CN110347482 B CN 110347482B
- Authority
- CN
- China
- Prior art keywords
- transaction
- queue
- newly arrived
- oltp
- existing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
OLTP affair combination rule and queue model improvement method based on OLTPShare, which belongs to the technical field of OLTP affair processing. The invention solves the problems that the transaction combination and processing efficiency of the existing OLTP transaction combination rule is low, and the queue model of the existing OLTPShare can not well cope with the continuous change of the workload. The invention abandons the mode of manually adding the transaction combination rule, provides the combination rule which is based on the machine learning clustering algorithm, can be automatically generated and dynamically adjusted during operation, and can improve the efficiency of transaction combination and processing by more than 50 percent; moreover, the present invention proposes an attachable and combinable queuing model that automatically adjusts parameter settings at runtime to account for potential workload variations. The invention can be applied to the technical field of OLTP transaction processing.
Description
Technical Field
The invention belongs to the technical field of OLTP Transaction Processing, and particularly relates to an OLTP (On-Line Transaction Processing) Transaction combination rule and queue model improvement method based On OLTPShare.
Background
The workload required for 89% of CMDBAC open source online items has very limited distinct declaration strings. Therefore, OLTP workload sharing has enormous application potential in reality.
OLTPShare (On-Line Transaction Processing Share, OLTP workload sharing technology) provides a batching strategy for OLTP workloads that queues all incoming transactions in an attempt to maximize their throughput. However, the transaction binding rules according to which the current version of the OLTPShare is determined before batch processing need to be manually established, which results in low efficiency of transaction binding and processing. Moreover, in practical applications, the queue model of OLTPShare does not cope well with the changing workload.
Disclosure of Invention
The invention aims to solve the problems that the existing OLTP transaction combination rule has low transaction combination and processing efficiency and the existing OLTPShare queue model cannot well cope with the constant workload change.
The technical scheme adopted by the invention for solving the technical problems is as follows: OLTP affair combination rule and queue model improvement method based on OLTPShare, the method includes the following steps:
step one, when a new transaction arrives, an SE thread checks whether the newly arrived transaction is a single declaration transaction, if the check result is that the newly arrived transaction is not the single declaration transaction, the newly arrived transaction is an incorporable transaction, and the transaction is executed one by one;
if the checking result is that the newly arrived transaction is a single declaration transaction, the SE thread inquires the combination rule of the existing queues and judges whether the newly arrived transaction is added into any one of the existing queues or not;
step two, if newly arrived transactions are added into the existing queue C, the newly arrived transactions and the transactions in the existing queue C are processed in batch;
if the newly arrived transaction is not added into any existing queue, the newly arrived transaction is used as a single new queue to be applied to the combination of the subsequently arrived transactions;
step three, when the affair A in a certain queue is being processed, detecting the arrival of the new added affair B in the queue, and checking whether the affair A is compatible with the affair B; if the transaction A is compatible with the transaction B, continuing to execute the step four, and if the transaction A is not compatible with the transaction B, continuing to execute the step five;
if the transaction A is compatible with the transaction B, processing the transaction A and the transaction B in parallel, and recording the initial position when the transaction B is added, if the initial position when the transaction B is added is not the head of the table of the transaction B, continuing to query from the head of the table of the transaction B when the tail of the table of the transaction B is reached until the recorded initial position of the transaction B is reached;
and step five, if the transaction A is incompatible with the transaction B, waiting for the transaction A to finish processing, and then starting to process the transaction B.
The invention has the beneficial effects that: the invention provides an OLTP transaction combination rule and queue model improvement method based on OLTPShare, abandons the mode of manually adding transaction combination rules, provides a combination rule which is based on a machine learning clustering algorithm, can be automatically generated and dynamically adjusted during operation, and can improve the efficiency of transaction combination and processing by more than 50%; moreover, the present invention proposes an attachable and combinable queuing model that automatically adjusts parameter settings at runtime to account for potential workload variations.
Drawings
FIG. 1 is a flow chart of the OLTP transaction binding rules and queue model improvement method based on OLTPShare of the present invention.
Detailed Description
The first embodiment is as follows: as shown in fig. 1, the OLTP transaction binding rule and queue model improvement method based on OLTPShare in this embodiment includes the following steps:
step one, when a new transaction arrives, checking whether the newly arrived transaction is a single-declaration transaction by an SE thread (an executor executing an SQL statement), and if the checking result is that the newly arrived transaction is not the single-declaration transaction, the newly arrived transaction is an incombinable transaction, and executing the transactions one by one;
executing transaction by transaction refers to implementing according to the traditional OLTP transaction processing method, namely executing one transaction;
if the checking result is that the newly arrived transaction is a single declaration transaction, the SE thread inquires the combination rule (merge rules) of the existing queue, and judges whether the newly arrived transaction is added into any existing queue;
the existing queue binding rules refer to: human-defined transaction binding rules. Whether a newly arrived transaction has a similar point with other transactions can be judged through a transaction combination rule, if so, the newly arrived transaction can be directly added into a queue similar to the newly arrived transaction to carry out batch processing together, and if not, the newly arrived transaction cannot be directly added into an existing queue to be executed independently;
step two, if newly arrived transactions are added into the existing queue C, the newly arrived transactions and the transactions in the existing queue C are processed in batch;
if the newly arrived transaction is not added into any existing queue, the newly arrived transaction is used as a single new queue to be applied to the combination of the subsequently arrived transactions;
if n (n is specified manually, e.g. 30, 40) consecutive transactions are determined not to belong to any of the existing queues, it can be determined that there is a problem with the binding rules of the existing queues and the binding rules need to be modified, and there are roughly two ways of modifying the binding rules:
1. the characteristics of the comparison are changed, for example from 5 to 3, or the number of characteristics is unchanged, but the characteristics of the comparison are changed, for example from (a, b, c) to (e, f, g).
2. The method of calculating the distance is changed, for example, from the euclidean distance to the manhattan distance, and so on.
The modified rules are applied to the binding of subsequently arriving new transactions.
The two modification modes can be adopted each time, or only one of the two modification modes can be adopted, and classification needs to be recalculated after each modification.
And according to the new characteristics of the continuously added transactions, the combination rule is continuously updated, and the processing speed of subsequent similar transactions is improved.
Step three, when the affair A in a certain queue is being processed, detecting the arrival of the new added affair B in the queue, and checking whether the affair A is compatible with the affair B; if the transaction A is compatible with the transaction B, continuing to execute the step four, and if the transaction A is not compatible with the transaction B, continuing to execute the step five;
if the transaction A is compatible with the transaction B, processing the transaction A and the transaction B in parallel, and recording the initial position when the transaction B is added, if the initial position when the transaction B is added is not the head of the table of the transaction B, continuing to query from the head of the table of the transaction B when the tail of the table of the transaction B is reached until the recorded initial position of the transaction B is reached;
for the table t of the transaction B, when the table is queried, the query is not started from the head of the table every time actually, and the query starting point is random, so that in order to traverse the whole table, the starting position of the table of the transaction B is recorded, and if the starting position is the head of the table, the whole table is traversed by directly traversing to the tail of the table. If the starting position is not the head of the table but the middle of the table, when traversing to the tail of the table, the front part of the starting point does not traverse, so the traversal is started from the head of the table until the starting position of the table of the transaction B.
And step five, if the transaction A is incompatible with the transaction B, waiting for the transaction A to finish processing, and then starting to process the transaction B.
The second embodiment is as follows: the present embodiment is a further limitation to the OLTP transaction combination rule and queue model improvement method based on oltpchare in the first embodiment, where the determining whether to add the newly arrived transaction to any existing queue includes the following specific processes:
and calculating the similarity between the newly arrived transaction and the existing transaction in the queue by a clustering method to judge whether the newly arrived transaction is added into any existing queue.
First, the transaction characteristics to be compared are set, for example, if the transactions are compared by 5 characteristics, then each transaction is represented by a five-dimensional vector, X ═ X1,x2,x3,x4,x5Then in the coordinate system, the transactions in each queue, must be very close together, and obvious clusters (cluster) can be seen. Artificially setting a y value representing the maximum allowable cluster radius, calculating the center of each cluster, calculating the distance between the new transaction and the center of each cluster (there are many calculation methods for the distance, such as euclidean distance, manhattan distance, etc.), and taking the cluster with the minimum distance, wherein the distance is less than y, which is the classification to which the newly arrived transaction belongs. Otherwise, if the distance is greater than y, which classification of the newly arrived transaction does not belong to, that is, the newly arrived transaction is not put into any existing queue and is executed independently.
The third concrete implementation mode: the present embodiment is further limited to the method for improving OLTP transaction combination rule and queue model based on OLTP share according to the first or second embodiment, where the checking, by the SE thread, whether the newly arrived transaction is a single declaration transaction is specifically:
checking whether the newly arrived transaction only consists of one SQL (structured Query language) statement through the SE thread, if so, determining that the newly arrived transaction is a single declaration transaction, otherwise, determining that the newly arrived transaction is not the single declaration transaction.
The fourth concrete implementation mode: the embodiment is a further limitation of the OLTP transaction combination rule and queue model improvement method based on OLTPShare according to the third embodiment, where the checking whether the transaction a and the transaction B are compatible specifically includes:
if the data involved by the transaction A and the transaction B are not conflicted, the transaction A is compatible with the transaction B, otherwise, the transaction A is incompatible with the transaction B.
Reading and writing the same data are conflicts, two writing operations write the same data as conflicts, reading the same data does not conflict, writing different data does not conflict, and reading and writing data is different and does not conflict.
The fifth concrete implementation mode: the present embodiment is a further limitation to the OLTP transaction combination rule and queue model improvement method based on OLTPShare described in the fourth embodiment, and in the third step, each queue is processed in parallel.
The above-described calculation examples of the present invention are merely to explain the calculation model and the calculation flow of the present invention in detail, and are not intended to limit the embodiments of the present invention. It will be apparent to those skilled in the art that other variations and modifications of the present invention can be made based on the above description, and it is not intended to be exhaustive or to limit the invention to the precise form disclosed, and all such modifications and variations are possible and contemplated as falling within the scope of the invention.
Claims (4)
1. OLTP affairs combination rule and queue model improvement method based on OLTPShare, which is characterized in that the method comprises the following steps:
step one, when a new transaction arrives, an SE thread checks whether the newly arrived transaction is a single declaration transaction, if the check result is that the newly arrived transaction is not the single declaration transaction, the newly arrived transaction is an incorporable transaction, and the transaction is executed one by one;
if the checking result is that the newly arrived transaction is a single declaration transaction, the SE thread inquires the combination rule of the existing queues and judges whether the newly arrived transaction is added into any one of the existing queues or not;
the specific process of judging whether to add the newly arrived transaction to any existing queue is as follows:
calculating the similarity between the newly arrived transaction and the existing transaction in the queue by a clustering method to judge whether the newly arrived transaction is added into any existing queue;
step two, if newly arrived transactions are added into the existing queue C, the newly arrived transactions and the transactions in the existing queue C are processed in batch;
if the newly arrived transaction is not added into any existing queue, the newly arrived transaction is used as a single new queue to be applied to the combination of the subsequently arrived transactions;
step three, when the affair A in a certain queue is being processed, detecting the arrival of the new added affair B in the queue, and checking whether the affair A is compatible with the affair B; if the transaction A is compatible with the transaction B, continuing to execute the step four, and if the transaction A is not compatible with the transaction B, continuing to execute the step five;
if the transaction A is compatible with the transaction B, processing the transaction A and the transaction B in parallel, and recording the initial position when the transaction B is added, if the initial position when the transaction B is added is not the head of the table of the transaction B, continuing to query from the head of the table of the transaction B when the tail of the table of the transaction B is reached until the recorded initial position of the transaction B is reached;
and step five, if the transaction A is incompatible with the transaction B, waiting for the transaction A to finish processing, and then starting to process the transaction B.
2. The OLTP transaction combination rule and queue model improvement method according to claim 1, wherein said SE thread checks whether a newly arrived transaction is a single declaration transaction, which is specifically:
and checking whether the newly arrived transaction only consists of one SQL statement or not through the SE thread, wherein if the newly arrived transaction only consists of one SQL statement, the newly arrived transaction is a single declaration transaction, and otherwise, the newly arrived transaction is not the single declaration transaction.
3. The OLTP transaction combination rule and queue model improvement method according to claim 2, wherein said checking whether transaction a and transaction B are compatible specifically comprises:
if the data involved by the transaction A and the transaction B are not conflicted, the transaction A is compatible with the transaction B, otherwise, the transaction A is incompatible with the transaction B.
4. The OLTP share based OLTP transaction binding rules and queue model improvement method of claim 3, wherein in step three, each queue is processed in parallel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910651419.6A CN110347482B (en) | 2019-07-18 | 2019-07-18 | OLTP transaction combination rule and queue model improvement method based on OLTPShare |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910651419.6A CN110347482B (en) | 2019-07-18 | 2019-07-18 | OLTP transaction combination rule and queue model improvement method based on OLTPShare |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110347482A CN110347482A (en) | 2019-10-18 |
| CN110347482B true CN110347482B (en) | 2021-07-23 |
Family
ID=68178919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910651419.6A Active CN110347482B (en) | 2019-07-18 | 2019-07-18 | OLTP transaction combination rule and queue model improvement method based on OLTPShare |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347482B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1928872A (en) * | 2005-09-09 | 2007-03-14 | 国际商业机器公司 | Device and method for writing data into disc by dynamic switching |
| CN107122354A (en) * | 2016-02-24 | 2017-09-01 | 华为技术有限公司 | Affairs perform method, apparatus and system |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
| CN109451076A (en) * | 2018-12-29 | 2019-03-08 | 乐蜜有限公司 | A kind of the merging treatment method, apparatus and electronic equipment of network request |
| CN109992359A (en) * | 2019-03-28 | 2019-07-09 | 深圳市创联时代科技有限公司 | A kind of transaction scheduling method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3079060B1 (en) * | 2015-04-08 | 2018-03-28 | Huawei Technologies Co., Ltd. | Load balancing for large in-memory databases |
-
2019
- 2019-07-18 CN CN201910651419.6A patent/CN110347482B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1928872A (en) * | 2005-09-09 | 2007-03-14 | 国际商业机器公司 | Device and method for writing data into disc by dynamic switching |
| CN107122354A (en) * | 2016-02-24 | 2017-09-01 | 华为技术有限公司 | Affairs perform method, apparatus and system |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
| CN109451076A (en) * | 2018-12-29 | 2019-03-08 | 乐蜜有限公司 | A kind of the merging treatment method, apparatus and electronic equipment of network request |
| CN109992359A (en) * | 2019-03-28 | 2019-07-09 | 深圳市创联时代科技有限公司 | A kind of transaction scheduling method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110347482A (en) | 2019-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10585889B2 (en) | Optimizing skewed joins in big data | |
| TW201833796A (en) | Method and apparatus for writing business data of block chain, and business subset determination method | |
| US9436739B2 (en) | Dynamic priority-based query scheduling | |
| US7546406B2 (en) | Virtualization of a global interrupt queue | |
| CN111294234B (en) | Parallel block chain fragmentation method based on intelligent contract optimization model | |
| US8397234B2 (en) | Determining a priority value for a thread for execution on a multithreading processor system | |
| US20090327220A1 (en) | Automated client/server operation partitioning | |
| US9003084B2 (en) | Sorting | |
| CN112114960B (en) | Scheduling strategy for remote sensing image parallel cluster processing adapting to internet scene | |
| CN111563093A (en) | Detection and avoidance system and method for union block chain conflict transaction | |
| CN112860670A (en) | Data migration method and device, electronic equipment and computer storage medium | |
| CN110347482B (en) | OLTP transaction combination rule and queue model improvement method based on OLTPShare | |
| CN111724256A (en) | Intelligent contract execution optimization method based on multi-core architecture | |
| CN108073658B (en) | Data synchronization system and method | |
| CN111782396A (en) | Concurrency flexible control method based on distributed database | |
| CN115496133A (en) | Density data stream clustering method based on self-adaptive online learning | |
| CN101699417B (en) | Interrupt method | |
| CN114020418A (en) | Particle swarm optimization virtual machine placement method containing roulette algorithm | |
| CN102063308A (en) | Method for controlling seismic exploration data processing flow | |
| Sasak-Okoń | Speculative query execution in Relational databases with Graph Modelling | |
| CN114385337B (en) | Task grouping scheduling method for distributed workflow system | |
| WO2024007922A1 (en) | Task migration method and apparatus, and device, storage medium and product | |
| JPH04355818A (en) | Data input/output controller | |
| US20220391448A1 (en) | Performance optimization of vector-based search and methods of using the same | |
| Jayasekara et al. | Communication Model for Parallel Iterative Stream Processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |