CN111767305A

CN111767305A - Self-adaptive database hybrid query method

Info

Publication number: CN111767305A
Application number: CN202010581766.9A
Authority: CN
Inventors: 文军; 郑立源; 李宇
Original assignee: University of Electronic Science and Technology of China
Current assignee: University of Electronic Science and Technology of China
Priority date: 2020-06-23
Filing date: 2020-06-23
Publication date: 2020-10-13
Anticipated expiration: 2040-06-23
Also published as: CN111767305B

Abstract

The invention discloses a self-adaptive database hybrid query method, which comprises the steps that a service process monitors a query request, calls a function of a database management layer to perform actual GPU query task execution and resource scheduling tasks, and returns a result to a query task client. Meanwhile, processing the subsequent request from the process, and transmitting the result back to the query task control process through an IPC mechanism; when the query control process completes the query, initiating an exit request to the server, and finishing the corresponding service thread by the service process; and the query control process generates a query plan, and a dynamic link library is used for intercepting a CUDA Runtime API initiated by the query task control process and converting the CUDA Runtime API into an IPC request of a service process. The method analyzes the query operation task on the hybrid platform based on the cost model according to the query data characteristics, creates a query optimization scheme, realizes database query by utilizing the cooperation of the CPU and the GPU, improves the overall query processing performance of the database, realizes the maximized utilization of the hardware performance of the computer, and reduces the extra time overhead.

Description

Self-adaptive database hybrid query method

Technical Field

The invention belongs to the technical field of GPU database query, and particularly relates to a database hybrid query method for realizing self-adaption through cooperation of a CPU and a GPU.

Background

The conventional GPU database is stored in a column mode, and is combined with a memory and a video memory, so that optimization is not needed, and an index is not needed. During production time, data are loaded into a video memory and a memory, and a disk is not required to be accessed, so that the overhead of accessing a sector is not required to be reduced by means of indexes. Through thousands of cores of each GPU, tens of thousands of threads are used for scanning the full table in parallel, and the method is particularly suitable for JOIN, fuzzy matching, Group By, full table scanning or aggregation and the like of tens of millions of lines. The GPU database uses the memory as a bridge to realize a three-level cache structure of a disk, the memory and a video memory, which can provide 10 times of CPU DRAM bandwidth and lower delay.

Current GPU database queries still suffer from some deficiencies:

1. each query task separately manages GPU resources, which can bring repeated overhead;

2. for the query of a large amount of data, the memory-video memory data exchange is frequently carried out by using PCie, different query tasks repeatedly transmit the same column storage data in a database, and the overall utilization rate of gpu is still low;

3. although hundreds or even thousands of stream processors in the GPU can provide powerful vector computing power, the efficiency of operations such as complex branch instructions, iterative processing, inter-thread data synchronization, large data high latency access, etc. is weaker than that of general purpose processors. Unfortunately, the relational operation model is not an ideal query processing model suitable for GPU vector computation features. The strategy adopted to cooperate with the CPU and the GPU is used for carrying out multi-branch statement query aiming at mass data, and the strategy is also one direction of current GPU database research;

4. for the same algorithm, since the GPU is weaker than the CPU in terms of logic control and complex data management capabilities, the overall performance of the GPU is not obvious to the advantage of the CPU algorithm for deep optimization, and even in some queries, the GPU algorithm performance is lower than the CPU algorithm. Therefore, the acceleration of the database by the GPU is not comprehensive, and the CPU and the GPU query processing module need to be combined according to the characteristics of the data and the characteristics of the operation.

Disclosure of Invention

Aiming at the defects in the prior art, the invention provides a self-adaptive database hybrid query method, so as to improve the overall query processing performance of the database, maximally utilize the performance of computer hardware and reduce the extra time overhead.

In order to achieve the purpose of the invention, the invention adopts the technical scheme that:

an adaptive database hybrid query method comprises the following steps:

s1, starting a database system service process and monitoring a query request;

s2, submitting an sql query statement to the database;

s3, generating a query plan according to the monitored sql query statement, and starting a query task control process;

s4, intercepting and inquiring a CUDA Runtime API initiated by a task control process by using a dynamic link library, and converting the CUDA Runtime API into an IPC request of a service process;

s5, after the server receives the IPC request, a new service thread is created to process the follow-up request from the process, and the result is transmitted back to the query task control process through the IPC mechanism;

and S6, when the query task control process completes the query, initiating an exit request to the server, and finishing the corresponding service thread by the service process.

Further, the step S1 specifically includes:

starting a database system service process, creating a CUDA context, modifying a folder pre-loaded column storage data to a system memory according to a starting parameter datadir, and then monitoring a query request.

Further, the step S3 specifically includes:

and generating a query plan by utilizing a query plan generation module based on a cost learning model according to the monitored sql query statement, setting the LD _ PRELOAD environment variable as a database dynamic link library path, executing the query plan, and starting a query task control process.

Further, the query plan generating module in step S3 specifically includes:

a self-adjusting execution time estimator for calculating estimated execution times of all available algorithms capable of performing the operation;

the algorithm selector is used for selecting an optimal algorithm according to the estimated execution time and the optimized exploratory formula;

and the hybrid query optimizer is used for constructing a physical query plan according to the logic query plan determined by the optimal algorithm and dispatching the database operation to the CPU and the GPU equipment.

Further, the step S3, according to the monitored sql query statement, generating a query plan based on the cost learning model by using a query plan generation module specifically includes the following sub-steps:

s31, for an operator OP (D, O) consisting of the data set D and the operation O, searching all available algorithms capable of executing the operation O by using an algorithm selector;

s32, calculating an estimated execution time ExeTime (D, A) for each algorithm A and each data set D by using a self-adjusting execution time estimator;

s33, selecting the algorithm with the shortest estimated execution time as the optimal algorithm by using the algorithm selector;

and S34, constructing a physical query plan by using the logic query plan determined by the hybrid query optimizer optimization algorithm, and dispatching the database operation to the CPU and the GPU equipment.

Furthermore, the query task control process combines the query plan generated by the query plan generation module with the database schema, compiles and forms a CPU host program for controlling data reading, and calls the CUDA Runtime API to control the flow of the whole query task according to the query plan.

Further, the step S4 specifically includes:

intercepting a CUDA Runtime API initiated by a query task control process by using a dynamic link library, using a globally unique CUDA Context through a service process, creating a single stream in a CUDA Context when the service process starts each query service thread to initiate GPU calls requested by the query process, and mapping the CUDA Runtime API calls contained in a CPU host program into IPC calls to the service process.

Further, the step S5 specifically includes:

after receiving the IPC request, the server establishes a new service process to process subsequent requests from the process, continuously analyzes the requests from the same query task process, executes responsive resource management and kernel calling operation through a database core management layer, and transmits the result back to the query task control process through an IPC mechanism.

Further, when different query processes use the same data, the service process determines that the PCie transmits the data to the GPU or directly uses the data stored in the memory of the GPU according to the working state of the system.

Further, the database core management layer comprises column storage data sharing, GPU hardware management and kernel calling logic.

The invention has the following beneficial effects:

(1) the invention uses a CPU and GPU mixed query mode to reduce Pcie bus data transmission pressure and enable computer hardware to be used with maximum efficiency;

(2) the invention uses the cost model based on learning, and can generate the execution plan according to the data characteristics and the operation without worrying about the problems that the model needs to be perfected and the administrator needs to maintain after the hardware resources are changed;

(3) the query process uses the dynamic link library to communicate with the database service process through an IPC mechanism, so that the coupling degree between the query task process and the database process is reduced, and the system has good expansibility;

(4) when different inquiry processes use the same data, the service process can determine whether the PCie transmits the data to the GPU or directly uses the data which is already stored in the memory of the GPU according to the working state of the system, so that the time waste caused by data transmission can be reduced.

Drawings

Fig. 1 is a schematic flow chart of the adaptive database hybrid query method of the present invention.

FIG. 2 is a schematic diagram of query plan generation according to the present invention;

FIG. 3 is a schematic diagram of a CPU and GPU hybrid query according to the present invention.

Detailed Description

The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.

As shown in fig. 1, an embodiment of the present invention provides an adaptive database hybrid query method, including the following steps S1 to S6:

s1, starting a database system service process and monitoring a query request;

in this embodiment, the present invention starts a database system service process, creates a CUDA context, and uses the CUDA context as a container to manage life cycles of all objects for calling the CUDA function; modifying the folder pre-loading column storage data to a system memory according to a starting parameter datadir, wherein the parameter datadir represents the position of the data to be loaded; and then listens for a Unix Domain socket to accept the query request.

S2, submitting an sql query statement to the database;

in the embodiment, since the conventional method uses the analysis model for cost evaluation, there are disadvantages that the model must be perfected after hardware resource changes, and the parameters thereof must be maintained and appropriately adjusted by a database administrator. Thus, the present invention uses a learning-based approach to estimate the execution time of operations on different processing devices. And mapping data characteristics (such as data size, data inclination and selectivity) and operations to obtain execution time by using the cost model as a decision basis, and distributing the database operations to the CPU and the GPU equipment according to the cost.

The method comprises the steps of generating a query plan based on a cost learning model by utilizing a query plan generation module according to monitored sql query statements, setting LD _ PRELOAD environment variables as database dynamic link library paths, executing the query plan, and starting a query task control process.

The cost learning model observes the correlation between the input data set and the result execution time and learns the correlation using a statistical method. The statistical method computes an approximation function after sufficient observations are collected, which selects the best algorithm for operation.

As shown in fig. 2, the query plan generating module specifically includes:

a self-adjusting execution time estimator S for calculating estimated execution times of all available algorithms capable of performing the operation, thereby providing an accurate and reliable database architecture and model-independent estimated execution times for the database algorithms. For the incoming algorithm a and data set D, the self-adjusting execution time estimator S computes an estimated execution time ExeTime (D, a).

The algorithm selector A is used for selecting an optimal algorithm according to the estimated execution time and the optimized exploratory formula; the algorithm selector a uses a self-adjusting execution time estimator S to obtain estimated execution times for all available algorithm execution operations. For the operator OP (D, O) consisting of the data set D and the operation O, the algorithm selector a finds all available algorithms for O and calculates the estimated execution time ExeTime (D, a) for them with the self-adjusting execution time estimator S, and then determines the optimal algorithm a opt. To optimize the response time, the system selects the algorithm that estimates the shortest execution time.

And the hybrid query optimizer H is used for constructing a physical query plan Q phy according to the logical query plan Q log determined by the optimal algorithm and dispatching the database operation to the CPU and the GPU equipment.

The generating of the query plan by the query plan generating module according to the monitored sql query statement based on the cost learning model specifically comprises the following steps:

The query task control process combines the query plan generated by the query plan generating module with a database schema (database object set), compiles and forms a CPU host program for controlling data reading, and the CPU host program calls a CUDA Runtime API to control the flow of the whole query task according to the query plan.

As shown in fig. 3, the query process in the present invention uses a dynamic link library to communicate with a database service process through an IPC (inter-process communication) mechanism, thereby reducing the coupling between the query task process and the database process. The system architecture is divided into two parts of a CPU and a GPU. The CPU is divided into database service process, query plan generating process and query process (query plan generating module). The GPU part mainly comprises a kernel (CUDA kernel function) implementation of an execution plan, and is loaded and called by a service process.

in the embodiment, the CUDARuntime API initiated by the task control process is intercepted and inquired by using the dynamic link library, so that a mechanism for triggering CUDA context created by a CUDA is avoided, and the time consumed by independently creating each inquiry process is saved by using the globally unique CUDA context through the service process. In order to avoid blocking caused by CUDA Runtime API calls between each query process, when each query service thread is started by a service process, a separate stream (GPU operation queue) is created in CUDAContext to initiate a GPU call requested by the query process. CUDAAPI calls carried out on different streams cannot be mutually blocked and can be called by a GPU (graphics processing unit) concurrently, so that PCie data transmission and kernel calls of different query tasks can be carried out. The CUDA Runtime API call contained in the CPU host program is then mapped as an IPC call to the service process.

in this embodiment, after receiving an IPC request, the server of the present invention creates a new service process to process subsequent requests from the process, continuously analyzes the requests from the same query task process, executes resource management and kernel call operations in response through the database core management layer, and returns the result to the query task control process through the IPC mechanism.

And the service process calls a function of the database management layer to perform actual GPU query task execution and resource scheduling tasks through an IPC request initiated by the dynamic link library, and returns the result to the query task client.

The database core management layer comprises logic of column storage data sharing, GPU hardware management, kernel calling and the like. Database management functions are provided to the service process in the form of a library of functions. The service process uses a multi-threaded architecture, and different query task requests respond using independent threads. Concurrent queries can be well supported. When different inquiry processes use the same data, the service process can determine whether the PCie transmits the data to the GPU or directly uses the data stored in the GPU memory according to the working state of the system. The time waste caused by data transmission can be reduced.

According to the method for realizing database query by the cooperative operation of the CPU and the GPU, the query operation tasks on the hybrid platform are analyzed based on the cost model according to the query data characteristics, the operation type, the PCIe channel data transmission performance between the GPU and the CPU, the GPU parallel computing performance and other related factors, so that a query optimization scheme is created, the overall query processing performance of the database is improved, the hardware performance of the computer is utilized to the maximum extent, and the extra time overhead is reduced.

It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Those skilled in the art can make various other specific changes and combinations based on the teachings of the present invention without departing from the spirit of the invention, and these changes and combinations are within the scope of the invention.

Claims

1. An adaptive database hybrid query method is characterized by comprising the following steps:

s1, starting a database system service process and monitoring a query request;

s2, submitting an sql query statement to the database;

2. The adaptive database hybrid query method according to claim 1, wherein the step S1 specifically includes:

3. The adaptive database hybrid query method according to claim 1, wherein the step S3 specifically includes:

4. The adaptive database hybrid query method according to claim 3, wherein the query plan generating module in step S3 specifically includes:

5. The adaptive database hybrid query method according to claim 4, wherein the step S3 is to generate a query plan based on the cost learning model by using a query plan generation module according to the monitored sql query statement, and specifically includes the following sub-steps:

6. The adaptive database hybrid query method according to claim 5, wherein the query task control process combines the query plan generated by the query plan generation module with the database schema, compiles and forms a CPU host program for controlling data reading, and calls a CUDA Runtime API to control the flow of the whole query task according to the query plan.

7. The adaptive database hybrid query method according to claim 6, wherein the step S4 specifically includes:

intercepting CUDA Runtime API initiated by a query task control process by using a dynamic link library, using a globally unique CUDA Context through a service process, creating a single stream in the CUDA Context when the service process starts each query service thread to initiate GPU call requested by the query process, and mapping CUDARuntime API call contained in a CPU host program into IPC call to the service process.

8. The adaptive database hybrid query method according to claim 1, wherein the step S5 specifically includes:

9. The adaptive database hybrid query method according to claim 8, wherein the service process determines that the PCie transmits data to the GPU or directly uses data already stored in a memory of the GPU according to a system working state when different query processes use the same data.

10. The adaptive database hybrid query method of claim 8, wherein the database core management layer comprises column store data sharing, GPU hardware management, and kernel call logic.