CN112363812B - Database connection queue management method based on task classification and storage medium - Google Patents
Database connection queue management method based on task classification and storage medium Download PDFInfo
- Publication number
- CN112363812B CN112363812B CN202011284651.XA CN202011284651A CN112363812B CN 112363812 B CN112363812 B CN 112363812B CN 202011284651 A CN202011284651 A CN 202011284651A CN 112363812 B CN112363812 B CN 112363812B
- Authority
- CN
- China
- Prior art keywords
- queue
- request
- task
- thread
- connection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
- G06F16/2456—Join operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a database connection queue management method based on task classification and a storage medium, belonging to the technical field of databases, aiming at solving the technical problem of how to solve the access performance problem of a database when facing massive concurrent connection, improving the operation efficiency of accessing the database in a thread pool connection management mode and ensuring the realization of more efficient and more refined database request queue scheduling, and the technical scheme is as follows: the method is based on a MySQL database, classifies client connection requests according to request types in a thread pool operation mode, and schedules different requests of a user in a reasonable and efficient mode to give the best operation practice to the user; the method comprises the following specific steps: s1, monitoring a data request; s2: request task grading: requesting a task queue and adjusting the dynamic priority; and S3, dynamically expanding and contracting the thread groups in parallel on site.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a database connection queue management method based on task classification and a storage medium.
Background
Under the current internet environment with mass flow, a database is used as bottom-layer data summarization and access support software of an application system, the high-efficiency throughput rate of the database is the basic requirement of database software and the continuously improved technical direction, and particularly, when the database is connected facing mass clients, how to schedule client requests in an optimal mode is a technical subject of the database software, so that the optimal performance of the database is achieved.
The traditional technology of the MySQL database is to allocate a thread to each connection request of a client, and for a MySQL instance, a thread is used to execute a statement in each connection of the client. However, when the number of concurrency exceeds a certain number, the total performance of the instance may be reduced as the number of connections continues to increase, and the reason for the performance reduction is mainly two points:
firstly, MySQL creates a thread for each connection request, and as the number of threads rises, frequent thread context switching is caused, and the cache hit rate of a CPU is reduced;
secondly, a large number of threads can cause the contention of the shared resources to be aggravated, and the lock acquisition for protecting the shared resources can become a great performance bottleneck.
Some improved versions of the MySQL database allow the server to maintain good performance even with a large number of client connections by introducing a dynamic thread pool. The server, by managing client connection requests using a thread pool, greatly reduces the number of concurrent threads, which reduces context switching and hot lock contention for threads. The use of the thread pool has the most obvious influence on OLTP workload (relatively short CPU constraint query), and in the known performance test, when mass data connection is faced, the performance index shown by the thread pool connection management model is obviously superior to that of a single-connection single-thread running mode.
Currently, the known thread pool connection management is implemented as two client connection request task queues, namely a high priority queue and a low priority queue, wherein the two queues queue respectively according to the state of the request task, and when the client request task is:
firstly, a client connection request is in a transaction;
the client connection request is holding a table lock;
the client side connection request table locking mode has effective value;
the client connects and requests to hold the metadata lock;
the client connection request is requesting a global read lock;
sixthly, the client connection request is requesting a backup lock.
In the above states, the task is added to the high priority queue, so that the client connects to the fast completion request and releases the lock of the occupied shared resource, while the ordinary client requests are added to the low priority queue by default. By scheduling two task queues with different priorities, the overall execution efficiency of the task queues is improved to a certain extent. When the MySQL database runs in a thread pool connection management mode, priority scheduling of client requests can be improved to a certain extent, but the MySQL database does not distinguish types of client connections, does not care about specific operation types of client requests, and cannot finely distinguish different types of operations which are short in time consumption, small in occupied shared resources, long in time consumption and large in occupied shared resources, so that the overall performance of the thread pool is not expressed in an optimal state.
In summary, there is no method for distinguishing the client connection requests according to the operation types to achieve the best performance in the prior art. Therefore, how to solve the problem of access performance when the database faces massive concurrent connection, improve the operating efficiency of accessing the database in a thread pool connection management mode, and ensure that more efficient and more precise scheduling of the database request queue is a problem to be solved urgently at present.
Disclosure of Invention
The technical task of the invention is to provide a database connection queue management method and a storage medium based on task classification, so as to solve the problem of access performance of a database when massive concurrent connections are faced.
The technical task of the invention is realized in the following way, a database connection queue management method based on task classification, the method is based on MySQL database, under the thread pool operation mode, the connection request of the client is classified according to the request type, and different requests of the user are dispatched in a reasonable and efficient way, so as to give the best operation practice to the user; the method comprises the following specific steps:
s1, monitoring a data request;
s2: request task grading: requesting a task queue and adjusting the dynamic priority;
and S3, dynamically expanding and contracting the thread groups in parallel on site.
Preferably, the data request monitoring in step S1 is as follows:
s101, loading a thread pool connection management function in a plug-in mode when a MySQL database is started, and waiting for connection and access of a client after a connection thread pool is started;
s102, connecting a thread pool to construct thread groups, wherein the number of the thread groups is consistent with the number of CPU cores of the running host, so that the maximum operational capability of a CPU is exerted;
s103, after the client is connected, obtaining a thread group number to be added to the connection through the digital-analog operation of the unique digital identifier connected with the client and a bus thread group, and adding the client to a corresponding thread group;
s104, distinguishing a management link and a data link;
s105, after the client side is connected and added into the task queue, the independent monitoring thread of the thread group starts to poll data requests connected with all the client sides in the thread group;
s106, after the monitoring thread reads the client connection request, judging whether the client connection is in one or more states of a transaction, a lock holding a shared resource and a global lock being requested according to the state of the client connection:
if yes, the priority of the data request is considered as high priority, and the data request is added into a high priority queue;
if not, adding other data requests into the common priority queue;
s107, judging whether the service is stopped:
if yes, ending;
if not, jumping to step S105.
Preferably, each thread group is provided with a maximum parallel thread number, and each thread group is allocated with an independent thread for monitoring data requests connected with all clients in the thread group.
Preferably, the distinguishing between the management link and the data link in step S104 is as follows:
s10401, adding the client connection to different task queues of the thread group according to the type of the client connection;
s10402, judging the connection type:
firstly, when the connection type is a management link, directly adding the link into a high-priority queue;
and secondly, when the connection type is a common data link, adding the link into a low-priority queue.
Preferably, the task queue request and the dynamic priority adjustment in step S2 are specifically as follows:
s201, a monitoring thread previews request data connected with all clients in a low-priority queue, only checks a data type field carried in a request data head, and does not read the request data;
s202, performing secondary classification on the request data according to the viewed request data types, wherein the secondary classification is divided into five operation types;
s203, judging whether the priority needs to be adjusted:
if yes, executing step S204;
if not, jumping to the step S205;
s204, dynamically adjusting the priority of the three types of secondary task queues according to different service scenes;
s205, judging whether the service is stopped:
if yes, ending;
if not, jumping to step S201.
Preferably, the five operation types in step S202 are specifically as follows:
(1) and management operation: operations such as query and modification (SET) variables, query running States (SHOW) and the like except for the following three types of operations, wherein the operations are generally operations with extremely low overhead, are executed firstly, and the data requests of the type are directly added into a high-priority queue;
(2) and transaction type operation: when the operation request is a transaction type operation, such as various DDL statements and definite transaction starting statements, the operation occupies more shared resources, but the occurrence frequency of the operation is low, and the execution of higher priority is ensured, and the operation is added into a transaction operation queue;
(3) and updating operation: when the operation request is an automatically submitted UPDATE (INSERT, UPDATE, DELETE) operation, such as most DML statements except SELECT, the type of statement will occupy relatively few shared resources, but the occurrence frequency is generally medium, and it should be ensured that the medium priority is executed, and the medium priority is added to the UPDATE operation queue;
(4) and query operation: when the operation type is a query type operation, such as a SELECT statement, the type statement occupies the least shared resources, but has the highest occurrence frequency and possibly longer execution time, and should ensure the execution with lower priority, add the query operation into a query operation queue;
(5) and long task operation: when the operation type is a task which consumes a long time, such as export (DUMP), Copy (CLONE) and the like, the statement of the type can monopolize one thread for a long time, the task is directly executed, the thread of the task is not counted in the number of concurrent threads of the thread group, and the quota of the normal number of working threads of the thread group is ensured not to be occupied.
Preferably, the priority of the transaction operation queue, the update operation queue and the query operation queue is transaction operation queue > update operation queue > query operation queue.
Preferably, in step S3, the parallel live dynamic scaling of the thread group is specifically as follows:
s301, after client request data are distributed to proper queues, each thread group immediately starts a working thread to process data requests according to priorities, and each task request in a high-priority queue is processed firstly;
s302, after the high-priority queue is processed, the thread group requests the priority to process the secondary task queue according to the preset tertiary queue;
s303, judging the number of waiting requests and the number of parallel threads of the task queue:
firstly, if the waiting number of the task queue is large and the duration time exceeds a threshold value, increasing the number of parallel threads of the thread group;
and secondly, if the waiting number of the task queue is small, the number of parallel threads is large, and the duration time exceeds a threshold value, recovering idle threads of the thread group, and reducing the number of the parallel threads.
An electronic device, comprising: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executes computer-executable instructions stored by the memory to cause the at least one processor to perform the method for task classification-based database connection queue management as described above.
A computer readable storage medium, having stored therein computer executable instructions, which when executed by a processor, implement a task classification based database connection queue management method as described above.
The database connection queue management method and the storage medium based on task classification have the following advantages that:
compared with the existing database thread pool connection management mode, the invention optimizes and improves the database from multiple angles, shows excellent performance indexes, solves the problem of access performance of the database in the face of massive concurrent connection, and improves the operation efficiency of accessing the database in the thread pool connection management mode;
secondly, the connection requests of the client are classified according to the request types based on the MySQL database in a thread pool operation mode, so that the overall throughput rate of the database is improved, different requests of users are dispatched in a reasonable and efficient mode, the best operation practice is provided for the users, optimization, promotion and transformation are performed on the original implementation, and the MySQL database has strong practicability and innovation;
the invention solves the problem of quick recovery when the database load is abnormal by distinguishing the client connection type and preferentially processing the management type connection link request;
by adopting the independent and stable monitoring thread analysis request mode when the client connection request arrives, the complexity and the performance overhead of switching the roles of the monitoring thread and the task processing thread are avoided, the monitoring thread is enabled to be dedicated to the classification of the client request task, and the problem that the database monitoring task cannot be processed in time at the first time is solved;
after the connection request of the client arrives, two levels of task queues of a high-priority queue and a low-priority queue and a queue classified according to operation types are adopted, so that the task request is more finely scheduled, and the running performance of a thread pool connection management mode is further improved;
the invention provides a solution for smoothly improving the performance in the time dimension and smoothly recycling idle resources by adopting a mode of time delay expansion and time delay contraction after the number of parallel connections of the thread pool reaches the existing limit, so that the performance accumulation performance of the thread pool connection management for a long time is further improved;
the invention ensures that the data request of the management operation is firstly processed and the high priority of the management operation by distinguishing the management link and the data link, thereby leading a database administrator to intervene in the operation of the database at the first time, further carrying out necessary adjustment on the operation state of the database and ensuring the healthy and efficient operation of the database;
according to the invention, through the independent monitoring thread, according to a single responsibility principle, the monitoring thread is only used for rapidly processing the data requests connected with all the clients in the thread group, so that the data requests connected with all the clients can be read and distributed to the task queue at the first time;
the invention distinguishes different request types through two stages of task queues, dynamically adjusts the priority of the three types of queues according to the operation scene and the service requirement of the user, and can finely adjust the performance of the database by a flexible and effective strategy so as to exert the optimal performance of the thread pool connection management;
tenth, the invention can rapidly cope with the sudden flow surge through the dynamic capacity expansion and reduction of the parallel thread number of the thread group, and timely release the resources in idle time, so that the whole performance of the database can be kept smooth and output.
Compared with the prior art, the invention provides a more excellent and efficient method for processing massive database connection requests.
Drawings
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a block diagram of a data request snooping process;
FIG. 2 is a flow diagram of a request task hierarchy process;
FIG. 3 is a flow diagram of parallel thread dynamic scaling.
Detailed Description
The task classification-based database connection queue management method and storage medium according to the present invention will be described in detail with reference to the accompanying drawings and specific embodiments.
Example 1:
the invention relates to a database connection queue management method based on task classification, which is based on a MySQL database, classifies client connection requests according to request types in a thread pool operation mode, and schedules different requests of users in a reasonable and efficient manner to give optimal operation practice to the users; the method comprises the following specific steps:
s1, monitoring a data request;
s2: request task grading: requesting a task queue and adjusting the dynamic priority;
and S3, dynamically expanding and contracting the thread groups in parallel on site.
As shown in fig. 1, the data request snooping in step S1 in this embodiment is as follows:
s101, loading a thread pool connection management function in a plug-in mode when a MySQL database is started, and waiting for connection and access of a client after a connection thread pool is started;
s102, connecting a thread pool to construct thread groups, wherein the number of the thread groups is consistent with the number of CPU cores of the running host, so that the maximum operational capability of a CPU is exerted; and each thread group is provided with a maximum parallel thread number, and is simultaneously allocated with an independent thread for monitoring data requests connected with all clients in the thread group.
S103, after the client is connected, obtaining a thread group number to be added to the connection through the digital-analog operation of the unique digital identifier connected with the client and a bus thread group, and adding the client to a corresponding thread group;
s104, distinguishing a management link and a data link; the method comprises the following specific steps:
s10401, adding the client connection to different task queues of the thread group according to the type of the client connection;
s10402, judging the connection type:
firstly, when the connection type is a management link, directly adding the link into a high-priority queue;
and secondly, when the connection type is a common data link, adding the link into a low-priority queue.
S105, after the client side is connected and added into the task queue, the independent monitoring thread of the thread group starts to poll data requests connected with all the client sides in the thread group;
s106, after the monitoring thread reads the client connection request, judging whether the client connection is in one or more states of a transaction, a lock holding a shared resource and a global lock being requested according to the state of the client connection:
if yes, the priority of the data request is considered as high priority, and the data request is added into a high priority queue;
if not, adding other data requests into a common priority queue;
s107, judging whether the service is stopped:
if yes, ending;
if not, jumping to step S105.
As shown in fig. 2, the task queue request and dynamic priority adjustment in step S2 in this embodiment are as follows:
s201, a monitoring thread previews request data connected with all clients in a low-priority queue, only checks a data type field carried in a request data head, and does not read the request data;
s202, performing secondary classification on the request data according to the viewed request data types, wherein the secondary classification is divided into five operation types; the method comprises the following specific steps:
(1) and management operation: operations such as query and modification (SET) variables, query running States (SHOW) and the like except for the following three types of operations, wherein the operations are generally operations with extremely low overhead, are executed firstly, and the data requests of the type are directly added into a high-priority queue;
(2) and transaction type operation: when the operation request is a transaction operation, such as various DDL statements and definite transaction starting statements, the operation occupies more shared resources, but the occurrence frequency of the operation is low, and the execution of higher priority is ensured, and the operation is added into a transaction operation queue;
(3) and updating operation: when the operation request is an automatically submitted UPDATE (INSERT, UPDATE, DELETE) operation, such as most DML statements except SELECT, the type of statement will occupy relatively few shared resources, but the occurrence frequency is generally medium, and it should be ensured that the medium priority is executed, and the medium priority is added to the UPDATE operation queue;
(4) and query operation: when the operation type is a query type operation, such as a SELECT statement, the type statement occupies the least shared resources, but has the highest occurrence frequency and possibly longer execution time, and should be executed with lower priority, and is added into a query operation queue;
(5) and long task operation: when the operation type is a task which consumes a long time, such as export (DUMP), Copy (CLONE) and the like, the statement of the type can monopolize one thread for a long time, then the task is directly executed, the thread of the task is not counted in the number of concurrent threads of the thread group, and the quota of the number of normal working threads of the thread group is ensured not to be occupied.
S203, judging whether the priority needs to be adjusted:
if yes, go to step S204;
if not, jumping to the step S205;
s204, dynamically adjusting the priority of the three types of secondary task queues according to different service scenes;
s205, judging whether the service is stopped:
if yes, ending;
if not, jumping to step S201.
In this embodiment, the priority is transaction operation queue > update operation queue > query operation queue.
As shown in fig. 3, in step S3, the details of the parallel field dynamic expansion of the thread group in the present embodiment are as follows:
s301, after client request data are distributed to proper queues, each thread group immediately starts a working thread to process data requests according to priorities, and each task request in a high-priority queue is processed firstly;
s302, after the high-priority queue is processed, the thread group requests the priority to process the secondary task queue according to the preset tertiary queue; because of the possibility of more queued data requests, each thread group allows a certain number of parallel threads to process multiple tasks simultaneously, but when the number of tasks backlogged in each thread group for a period of time is too large, the integrity of the thread pool is affected. If the thread group is always in the full-bin running state, after a period of time, the maximum parallel thread number is dynamically expanded to process more tasks in parallel after the preset threshold value is exceeded.
S303, judging the number of waiting requests and the number of parallel threads of the task queue:
firstly, if the waiting number of the task queue is large and the duration time exceeds a threshold value, increasing the number of parallel threads of the thread group;
and secondly, if the waiting number of the task queue is small, the number of the parallel threads is large, and the duration time exceeds a threshold value, recovering idle threads of the thread group, and reducing the number of the parallel threads.
Example 2:
an embodiment of the present invention further provides an electronic device, including: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform a method of database connection queue management based on task classification as in any one of the embodiments.
Example 3:
the embodiment of the invention also provides a computer-readable storage medium, wherein a plurality of instructions are stored, and the instructions are loaded by the processor, so that the processor executes the database connection queue management method based on task classification in any embodiment of the invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R database connection queue management method based on task classification and storage medium M, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (7)
1. A database connection queue management method based on task classification is characterized in that the method is based on a MySQL database, classifies client connection requests according to request types in a thread pool operation mode, and schedules different requests of users in a reasonable and efficient manner to give optimal operation practices to the users; the method comprises the following specific steps:
s1, data request monitoring, which is specifically as follows:
distinguishing the management link from the data link specifically as follows:
adding the client connection to different task queues of the thread group according to the type of the client connection;
judging the connection type:
when the connection type is a management link, directly adding the link into a high-priority queue;
secondly, when the connection type is a common data link, adding the link into a low-priority queue;
after the monitoring thread reads the client connection request, judging whether the client connection is in one or more states of a transaction, a lock holding a shared resource and a global lock being requested according to the state of the client connection:
if yes, the priority of the data request is considered as high priority, and the data request is added into a high priority queue;
if not, adding the data request to a low-priority queue;
s2: request task grading: requesting a task queue and adjusting the dynamic priority; the method comprises the following specific steps:
s201, a monitoring thread previews request data connected with all clients in a low-priority queue, only checks a data type field carried in a request data head, and does not read the request data;
s202, performing secondary classification on the request data according to the viewed request data types, wherein the secondary classification is divided into five operation types; the five types of operation types are management type operation, transaction type operation, updating type operation, query type operation and long task operation respectively; wherein the management class operates: the operation with low overhead is executed firstly, and the data request is directly added into a high-priority queue; long task operation: when the operation type is a task which consumes long time for exporting and copying, the statement of the type can monopolize a thread for a long time, then the task is directly executed, and the thread of the task is not counted in the number of concurrent threads of a thread group;
s203, judging whether the priority needs to be adjusted:
if yes, executing step S204;
if not, jumping to the step S205;
s204, dynamically adjusting the priority of the three types of secondary task queues according to different service scenes; the three types of secondary task queues refer to transaction type operation, updating type operation and query type operation; wherein the transaction class operation: when the operation request is a transaction operation, ensuring high-priority execution with an explicit transaction starting statement, and adding the operation request into a transaction operation queue; updating class operation: when the operation request is an updating operation which is automatically submitted, the execution of medium priority is ensured, and the operation request is added into an updating operation queue; query class operation: when the operation type is query operation, ensuring low-priority execution, and adding the operation type into a query operation queue;
s205, judging whether the service is stopped:
if yes, ending;
if not, jumping to the step S201;
s3, dynamically stretching thread groups in parallel on site, specifically as follows:
if the waiting number of the task queue is large and the duration time exceeds a threshold value, increasing the number of parallel threads of the thread group;
if the waiting number of the task queue is small, the number of the parallel threads is large, and the duration time exceeds a threshold value, the idle threads of the thread group are recovered, and the number of the parallel threads is reduced.
2. The method for managing the database connection queue based on task classification as claimed in claim 1, wherein the data request snooping in step S1 is as follows:
s101, loading a thread pool connection management function in a plug-in mode when a MySQL database is started, and waiting for connection and access of a client after a connection thread pool is started;
s102, connecting a thread pool to construct thread groups, wherein the number of the thread groups is consistent with the number of CPU cores of the running host, so that the maximum computing capability of a CPU is exerted;
s103, after the client is connected, obtaining a thread group number to be added to the connection through the digital-analog operation of the unique digital identifier connected with the client and a bus thread group, and adding the client to a corresponding thread group;
s104, distinguishing a management link and a data link;
s105, after the client side is connected and added into the task queue, the independent monitoring thread of the thread group starts to poll data requests connected with all the client sides in the thread group;
s106, after the monitoring thread reads the client connection request, judging whether the client connection is in one or more states of a transaction, a lock holding a shared resource and a global lock being requested according to the state of the client connection:
if yes, the priority of the data request is considered as high priority, and the data request is added into a high priority queue;
if not, adding the data request to a low-priority queue;
s107, judging whether the service is stopped:
if yes, ending;
if not, jumping to step S105.
3. The method as claimed in claim 2, wherein each thread group has a maximum parallel thread count, and each thread group is assigned an independent thread for monitoring data requests connected to all clients in the thread group.
4. The method for managing database connection queues according to claim 1, wherein the priorities of the transaction operation queue, the update operation queue and the query operation queue are transaction operation queue > update operation queue > query operation queue.
5. The task classification-based database connection queue management method according to claim 1, wherein in step S3, the thread group parallel field dynamic scaling is specifically as follows:
s301, after the client request data are distributed to proper queues, each thread group immediately starts a working thread to process the data request according to the priority, and each task request in the high-priority queue is processed firstly;
s302, after the high-priority queue is processed, the thread group processes a secondary task queue according to a preset three-type queue request priority;
s303, judging the number of waiting requests and the number of parallel threads of the task queue:
firstly, if the waiting number of the task queue is large and the duration time exceeds a threshold value, increasing the number of parallel threads of the thread group;
and secondly, if the waiting number of the task queue is small, the number of the parallel threads is large, and the duration time exceeds a threshold value, recovering idle threads of the thread group, and reducing the number of the parallel threads.
6. An electronic device, comprising: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method for task classification-based database connection queue management of any of claims 1-5.
7. A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the method for task classification-based database connection queue management according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011284651.XA CN112363812B (en) | 2020-11-17 | 2020-11-17 | Database connection queue management method based on task classification and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011284651.XA CN112363812B (en) | 2020-11-17 | 2020-11-17 | Database connection queue management method based on task classification and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112363812A CN112363812A (en) | 2021-02-12 |
| CN112363812B true CN112363812B (en) | 2022-09-06 |
Family
ID=74515798
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011284651.XA Active CN112363812B (en) | 2020-11-17 | 2020-11-17 | Database connection queue management method based on task classification and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112363812B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988416B (en) * | 2021-03-02 | 2023-06-30 | 杭州迪普科技股份有限公司 | Message processing method, device, equipment and computer readable storage medium |
| CN114721834B (en) * | 2022-05-19 | 2022-08-26 | 国汽智控(北京)科技有限公司 | Resource allocation processing method, device, equipment, vehicle and medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899274A (en) * | 2015-05-27 | 2015-09-09 | 北方信息控制集团有限公司 | High-efficiency remote in-memory database access method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8037250B1 (en) * | 2004-12-09 | 2011-10-11 | Oracle America, Inc. | Arbitrating cache misses in a multithreaded/multi-core processor |
| US11630832B2 (en) * | 2017-06-07 | 2023-04-18 | Sap Se | Dynamic admission control for database requests |
| CN107688607A (en) * | 2017-07-26 | 2018-02-13 | 努比亚技术有限公司 | The method and mobile terminal of a kind of database access, computer-readable recording medium |
| CN110837401A (en) * | 2018-08-16 | 2020-02-25 | 苏宁易购集团股份有限公司 | Hierarchical processing method and device for java thread pool |
| CN111104210A (en) * | 2019-11-26 | 2020-05-05 | 苏宁云计算有限公司 | Task processing method and device and computer system |
| CN111813554A (en) * | 2020-07-17 | 2020-10-23 | 济南浪潮数据技术有限公司 | Task scheduling processing method and device, electronic equipment and storage medium |
-
2020
- 2020-11-17 CN CN202011284651.XA patent/CN112363812B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899274A (en) * | 2015-05-27 | 2015-09-09 | 北方信息控制集团有限公司 | High-efficiency remote in-memory database access method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112363812A (en) | 2021-02-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6411982B2 (en) | Thread based governor for time scheduled process execution | |
| CN111124679B (en) | Multi-source heterogeneous mass data-oriented time-limited automatic processing method | |
| US8041707B2 (en) | Database management system and method capable of dynamically generating new tasks that can be processed in parallel | |
| CA2785398C (en) | Managing queries | |
| US8818989B2 (en) | Memory usage query governor | |
| CN109828833B (en) | Queuing system and method for neural network training task | |
| US7441240B2 (en) | Process scheduling apparatus, process scheduling method, program for process scheduling, and storage medium recording a program for process scheduling | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| CN102096603B (en) | Task decomposition control method in MapReduce system and scheduling node equipment | |
| US8516488B1 (en) | Adjusting a resource estimate in response to progress of execution of a request | |
| CN112363812B (en) | Database connection queue management method based on task classification and storage medium | |
| US8583608B2 (en) | Maximum allowable runtime query governor | |
| US9836516B2 (en) | Parallel scanners for log based replication | |
| CN107273200B (en) | Task scheduling method for heterogeneous storage | |
| TWI394074B (en) | Methods, apparatus and computer programs for managing access to storage | |
| JP2008152618A (en) | Job assignment program, method and device | |
| US11561929B2 (en) | Method, device and computer program product for shrinking storage space | |
| CN113342477A (en) | Container group deployment method, device, equipment and storage medium | |
| US9996470B2 (en) | Workload management in a global recycle queue infrastructure | |
| CN105095495A (en) | Distributed file system cache management method and system | |
| CN116089414B (en) | Time sequence database writing performance optimization method and device based on mass data scene | |
| CN110413689B (en) | Multi-node data synchronization method and device for memory database | |
| EP3293625B1 (en) | Method and device for accessing file, and storage system | |
| CN113901141B (en) | Distributed data synchronization method and system | |
| Luo et al. | On performance stability in LSM-based storage systems (extended version) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |