CN117971906B - Multi-card collaborative database query method, device, equipment and storage medium - Google Patents
Multi-card collaborative database query method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117971906B CN117971906B CN202410389664.5A CN202410389664A CN117971906B CN 117971906 B CN117971906 B CN 117971906B CN 202410389664 A CN202410389664 A CN 202410389664A CN 117971906 B CN117971906 B CN 117971906B
- Authority
- CN
- China
- Prior art keywords
- card
- heterogeneous
- acceleration
- processing
- sql query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012545 processing Methods 0.000 claims abstract description 168
- 230000001133 acceleration Effects 0.000 claims abstract description 134
- 230000006870 function Effects 0.000 claims description 44
- 230000008569 process Effects 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 15
- 238000001514 detection method Methods 0.000 claims description 9
- 230000003068 static effect Effects 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 claims 4
- 238000013461 design Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 230000002776 aggregation Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 4
- 238000012163 sequencing technique Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000004888 barrier function Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000013078 crystal Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Abstract
The application discloses a multi-card collaborative database query method, a device, equipment and a storage medium, relating to the field of database design, comprising the following steps: when receiving an SQL query instruction sent by a client through a CPU of a database server, judging whether the estimated processing time of the SQL query instruction is smaller than a preset time threshold, if so, determining a target heterogeneous acceleration card for processing the instruction by detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card; the CPU is used for sending the data to be processed and the target function control parameters obtained after the SQL query instruction is analyzed to the target heterogeneous accelerator card; performing data processing by using the objective function control parameters through the objective heterogeneous acceleration card; and after the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, acquiring a processing result to complete data query operation. And the database query efficiency is effectively improved.
Description
Technical Field
The present invention relates to the field of database design, and in particular, to a method, an apparatus, a device, and a storage medium for querying a multi-card collaborative database.
Background
At present, with the continuous advancement of the artificial intelligence field and the information society, the variety and the installed capacity of databases in various industries are increased, the added silicon chips approach the limit of physical and economic cost, the CPU computing power is increased slightly, the traditional general computing system based on the von Neumann architecture can not meet the increasing high-performance data processing requirement, and the influence caused by the performance barrier problem of database software is more severe.
Disclosure of Invention
Accordingly, the present invention aims to provide a multi-card collaborative database query method, device, equipment and storage medium, which can effectively improve database performance and thus database query efficiency. The specific scheme is as follows:
In a first aspect, the present application provides a multi-card collaborative database query method, including:
When receiving an SQL query instruction sent by a client through a CPU of a database server, determining estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm;
Judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction;
the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed are sent to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface;
Processing the data to be processed by the target heterogeneous acceleration card by utilizing the target function control parameters so as to obtain a corresponding processing result;
After the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card is acquired through the corresponding PCIE interface so as to complete corresponding data query operation.
Optionally, the determining, based on a preset time prediction algorithm, the predicted processing time corresponding to the SQL query instruction includes:
And determining the estimated processing time corresponding to the SQL query instruction based on the writing stage time overhead operator, the reading stage time overhead multiplier, the query operation stage time overhead operator and the preset weight information corresponding to the client.
Optionally, after the determining whether the estimated processing time is less than the corresponding preset time threshold, the method further includes:
and if the estimated processing time is not smaller than the corresponding preset time threshold, processing the SQL query instruction through the CPU.
Optionally, if the step of detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card by a preset acceleration card scheduling rule, determining a target heterogeneous acceleration card for processing the SQL query instruction includes:
If yes, task information corresponding to the SQL query instruction is sent to a preconfigured heterogeneous task execution list;
Sequentially detecting the working states of the heterogeneous accelerator cards connected with the database server through a preset accelerator card scheduling rule;
when the working state of the current heterogeneous accelerator card is detected to be a busy state, judging whether the working state of a target accelerator core corresponding to the SQL query instruction on an FPGA chip on the current heterogeneous accelerator card is an idle state or not based on the task information, and obtaining a corresponding judging result;
If the judging result is negative, updating the current heterogeneous accelerator card, and detecting the working state of the new current heterogeneous accelerator card until the target heterogeneous accelerator card for processing the SQL query instruction is determined.
Optionally, the method further comprises:
when the working state of the current heterogeneous accelerator card is detected to be an idle state, the current heterogeneous accelerator card is used as a target heterogeneous accelerator card for processing the SQL query instruction;
And when the judgment result is yes, taking the current heterogeneous accelerator card as a target heterogeneous accelerator card for processing the SQL query instruction.
Optionally, the sending, by the CPU and the PCIE interface, the to-be-processed data and the objective function control parameters obtained after the parsing of the SQL query instruction to the target heterogeneous accelerator card includes:
Sending the data to be processed obtained after analyzing the SQL query instruction to the DDR4 memory of the target heterogeneous accelerator card through the CPU and the corresponding PCIE interface;
The CPU and the corresponding PCIE interface send target function control parameters obtained after analyzing the SQL query instruction to a static area of the target heterogeneous accelerator card; wherein, the objective function control parameters comprise data bit width and total data processing amount.
Optionally, the processing, by the target heterogeneous acceleration card, the data to be processed by using the target function control parameter to obtain a corresponding processing result includes:
And in the process of processing the data to be processed through the target heterogeneous accelerator card, transmitting the target function control parameters stored in the static area to corresponding acceleration cores so that the acceleration cores process the data to be processed, which are acquired from the DDR4 memory through an AXI4 bus, based on the target function control parameters.
In a second aspect, the present application provides a multi-card collaborative database query apparatus, including:
The time estimation module is used for determining the estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm when the SQL query instruction sent by the client is received through the CPU of the database server;
The acceleration card detection module is used for judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction;
The data sending module is used for sending the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed to the objective heterogeneous acceleration card through the CPU and the corresponding PCIE interface;
the data processing module is used for processing the data to be processed by utilizing the objective function control parameters through the objective heterogeneous acceleration card so as to obtain corresponding processing results;
And the processing result acquisition module is used for acquiring the processing result stored in the DDR4 memory on the target heterogeneous accelerator card through the corresponding PCIE interface after the CPU receives the processing completion signal sent by the target heterogeneous accelerator card, so as to complete corresponding data query operation.
In a third aspect, the present application provides an electronic device, comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the steps of the multi-card collaborative database query method.
In a fourth aspect, the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the multi-card collaborative database query method described above.
In the application, when the CPU of the database server receives the SQL query instruction sent by the client, the estimated processing time corresponding to the SQL query instruction is determined based on a preset time estimation algorithm; judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction; the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed are sent to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface; processing the data to be processed by the target heterogeneous acceleration card by utilizing the target function control parameters so as to obtain a corresponding processing result; after the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card is acquired through the corresponding PCIE interface so as to complete corresponding data query operation. In other words, when the SQL query instruction is received, firstly, the processing time of the instruction is estimated, then, whether the estimated processing time is smaller than a preset time threshold value is judged, and when the estimated processing time is smaller than the preset time threshold value, the target heterogeneous acceleration cards for processing the instruction are distributed by detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card. And then processing the data to be processed based on the received objective function control parameters through the objective heterogeneous accelerator card, and sending a corresponding processing completion signal to the CPU after the processing is completed. Thus, the database performance can be effectively improved, and the database query efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a multi-card collaborative database query method provided by the application;
FIG. 2 is a schematic diagram of a architecture of a specific FPGA-based multi-card collaboration database;
FIG. 3 is a flowchart of a specific multi-card collaborative database query method provided by the present application;
Fig. 4 is a schematic structural diagram of a multi-card collaborative database query device provided by the present application;
Fig. 5 is a block diagram of an electronic device according to the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
At present, with the continuous advancement of the artificial intelligence field and the information society, the variety and the installed capacity of databases in various industries are increased, the added silicon chips approach the limit of physical and economic cost, the CPU computing power is increased slightly, the traditional general computing system based on the von Neumann architecture can not meet the increasing high-performance data processing requirement, and the influence caused by the performance barrier problem of database software is more severe. Therefore, the application provides a multi-card collaborative database query scheme based on the FPGA, which can effectively improve the database performance, thereby improving the database query efficiency.
Referring to fig. 1, the embodiment of the invention discloses a multi-card collaborative database query method, which comprises the following steps:
and S11, when receiving an SQL query instruction sent by a client through a CPU of a database server, determining the estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm.
Referring to fig. 2, the multi-card collaborative database query acceleration architecture in this embodiment includes a corresponding database server, a plurality of heterogeneous acceleration cards (eight blocks can be deployed at most) based on FPGAs (Field Programmable GATE ARRAY, field-editable gate arrays), and PCIE (PERIPHERAL COMPONENT INTERCONNECT EXPRESS, high-speed serial computer expansion bus standard) interfaces connecting the two. The architecture adopts a hardware design mode of carrying a plurality of heterogeneous accelerator cards on a single database server so as to reduce the query processing delay of high-concurrency large-scale data to the greatest extent. The database server is a core of a multi-card collaborative database query architecture, can independently complete all data storage read-write and complex data processing tasks, and simultaneously, overall manages and collaborative scheduling work of all heterogeneous acceleration cards according to received query acceleration tasks, namely SQL (Structured Query Language ) query instructions. The heterogeneous acceleration card comprises corresponding FPGA chips (all acceleration cores required by executing the acceleration processing of the database are deployed on the FPGA chips, such as sequencing, aggregation, filtering, connection, grouping and the like), DDR4 (Double Data Rate 4, fourth generation Double Data Rate technology) memory, PCIE interfaces, key keyboards, crystal oscillator modules, power supply modules and JTAG (Joint Test Action Group, joint test action groups), which are an international standard test protocol and are mainly used for testing the inside of the chip, simulating and debugging the system, debugging ports, necessary resistance and capacitance and the like, and are responsible for processing query acceleration tasks sent by the connected database server so as to output processing results to a server user side. In this way, the database performance high-speed judging card based on the FPGA is connected with a CPU (central processing unit ) of the database server through the PCIE high-speed transmission interface, so that high-speed data exchange can be realized.
Furthermore, in this embodiment, according to the design of the multi-card collaborative acceleration architecture, one database server node typically deploys a plurality of heterogeneous acceleration cards, and different heterogeneous acceleration cards can implement two levels of SQL query parallelism between devices and within the device through a multi-card collaborative scheduling policy, so as to ensure that resources of the device can be fully utilized, furthest improve the computation throughput rate of concurrent tasks, and greatly reduce the SQL query latency loss. The multi-card collaborative scheduling strategy maintains a device list through an OpenCL (Open Computing Language ) framework, and develops and designs according to load balancing and task allocation modes.
Considering that the difference of data volume and the change of the working state of the acceleration card can influence the acceleration effect of the heterogeneous system on the query task when SQL query is executed. That is, if there are many concurrent inquiry tasks in a single period, a plurality of heterogeneous acceleration cards are in a busy state for a long time, and the inquiry tasks are queued up to cause an excessively long waiting time, so that effective acceleration cannot be formed. Therefore, the present embodiment designs a query task management manner based on time estimation, estimates task processing time of a received SQL query instruction in advance, and determines how to process the instruction according to the estimation result. That is, referring to fig. 3, in this embodiment, determining, based on the preset time prediction algorithm, the predicted processing time corresponding to the SQL query instruction includes: and determining the estimated processing time corresponding to the SQL query instruction based on the writing stage time overhead operator, the reading stage time overhead multiplier, the query operation stage time overhead operator and the preset weight information corresponding to the client. Regarding the preset time estimation algorithm, the following expression exists:
；
；
；
；
Wherein Tac is the estimated result of the inquiry time cost; x k,m、yk,m、zk,m is a write phase time overhead operator, a read phase time overhead multiplier, and a query operation phase time overhead operator, respectively; alpha, beta and gamma are preset weights of users, namely the preset weight information; n is the current query sequence number, k is the number of query packets; A. b is the maximum value of the written and read data quantity respectively; m is the maximum value of the memory of the accelerator card; m is the current writing data quantity; Writing a data overhead operator for the current query; x avg is a historical average write data overhead operator; Reading out a data overhead operator for the current query; y avg is a historical average readout data overhead operator; the sum of the operation complexity involved in the query task of the SQL query instruction is the current; i is the total number of operations of the task in the current query instruction; compplex sum is the sum of all task complexity. That is, in this embodiment, according to heterogeneous acceleration execution of different data volumes of different query operators, time costs of different stages of data writing, execution, reading and the like are analyzed to estimate the overall time costs of processing time of different data volumes of different SQL query requirements.
And step S12, judging whether the estimated processing time is smaller than a corresponding preset time threshold, if so, detecting the working states of each heterogeneous accelerator card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous accelerator card through a preset accelerator card scheduling rule, and determining the target heterogeneous accelerator card for processing the SQL query instruction.
Specifically, in this embodiment, as shown in fig. 3, if so, detecting, by a preset accelerator card scheduling rule, a working state of each of the heterogeneous accelerator cards currently connected to the database server and each of the accelerator cores on the FPGA chip on each of the heterogeneous accelerator cards, and determining a target heterogeneous accelerator card for processing the SQL query instruction includes: if yes, task information corresponding to the SQL query instruction is sent to a preconfigured heterogeneous task execution list; sequentially detecting the working states of the heterogeneous accelerator cards connected with the database server through a preset accelerator card scheduling rule; when the working state of the current heterogeneous accelerator card is detected to be a busy state, judging whether the working state of a target accelerator core corresponding to the SQL query instruction on an FPGA chip on the current heterogeneous accelerator card is an idle state or not based on the task information, and obtaining a corresponding judging result; if the judging result is negative, updating the current heterogeneous accelerator card, and detecting the working state of the new current heterogeneous accelerator card until the target heterogeneous accelerator card for processing the SQL query instruction is determined. Specifically, each heterogeneous accelerator card may be sequentially detected based on a pre-configured number. The preset time threshold value can be configured and correspondingly adjusted in advance based on actual requirements, and a corresponding time threshold value table can be specifically configured in advance.
Further, in this embodiment, the method further includes: when the working state of the current heterogeneous accelerator card is detected to be an idle state, the current heterogeneous accelerator card is used as a target heterogeneous accelerator card for processing the SQL query instruction; and when the judgment result is yes, taking the current heterogeneous accelerator card as a target heterogeneous accelerator card for processing the SQL query instruction.
It can be understood that, in this embodiment, when the amount of data to be processed of the query instruction is too low, the proportion of the transmission time loss of data between the CPU and the heterogeneous accelerator card is very large, and the efficiency of using the heterogeneous accelerator card is far lower than that of directly processing the database server under the same condition, that is, if the estimated processing time is not less than the corresponding preset time threshold, the SQL query instruction is processed by the CPU.
And step S13, sending the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface.
In this embodiment, the sending, by the CPU and the PCIE interface, the to-be-processed data and the objective function control parameter obtained after the parsing of the SQL query instruction to the objective heterogeneous accelerator card includes: sending the data to be processed obtained after analyzing the SQL query instruction to the DDR4 memory of the target heterogeneous accelerator card through the CPU and the corresponding PCIE interface; the CPU and the corresponding PCIE interface send the objective function control parameters obtained after analyzing the SQL query instruction to the static area (STATIC SHELL) of the objective heterogeneous accelerator card; wherein, the objective function control parameters comprise data bit width and total data processing amount. Specifically, after receiving the SQL query instruction, the CPU performs instruction parsing and optimization, and then invokes an API (Application Programming Interface, i.e., application programming interface) function corresponding to an acceleration kernel required for processing the instruction. Meanwhile, the CPU obtains corresponding data to be processed from a main memory in the database server by analyzing the SQL query instruction, and sends the data to be processed to the DDR4 memory of the determined target heterogeneous accelerator card in a PCIE+DMA (DYNAMIC MECHANICAL ANALYSIS, direct memory access) mode.
And step S14, processing the data to be processed by the target heterogeneous accelerator card by utilizing the target function control parameters so as to obtain a corresponding processing result.
Specifically, in this embodiment, the processing, by the target heterogeneous acceleration card, the data to be processed by using the target function control parameter to obtain a corresponding processing result includes: and in the process of processing the data to be processed through the target heterogeneous accelerator card, transmitting the target function control parameters stored in the static area to corresponding acceleration cores so that the acceleration cores process the data to be processed, which are acquired from the DDR4 memory through an AXI4 (Advanced eXtensible Interface, fourth generation advanced extensible interface) bus, based on the target function control parameters. And then, rewriting the obtained processing result into the DDR4 memory, and sending a corresponding processing completion signal to the CPU.
Step S15, after the CPU receives the processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card is obtained through the corresponding PCIE interface, so as to complete a corresponding data query operation.
Specifically, in this embodiment, when the CPU reads back the processing result from the DDR4 memory, pcie+dma is also used. And then, returning the processing result to the user query terminal, namely the client.
Therefore, in the embodiment of the application, when the SQL query instruction sent by the client is received through the CPU of the database server, the estimated processing time corresponding to the SQL query instruction is determined based on the preset time estimation algorithm; judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction; the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed are sent to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface; processing the data to be processed by the target heterogeneous acceleration card by utilizing the target function control parameters so as to obtain a corresponding processing result; after the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card is acquired through the corresponding PCIE interface so as to complete corresponding data query operation. In other words, when the SQL query instruction is received, firstly, the processing time of the instruction is estimated, then, whether the estimated processing time is smaller than a preset time threshold value is judged, and when the estimated processing time is smaller than the preset time threshold value, the target heterogeneous acceleration cards for processing the instruction are distributed by detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card. And then processing the data to be processed based on the received objective function control parameters through the objective heterogeneous accelerator card, and sending a corresponding processing completion signal to the CPU after the processing is completed. Thus, the database performance can be effectively improved, and the database query efficiency is improved.
The technical solution of the embodiment of the present application is specifically described below with reference to the schematic diagram of the database architecture disclosed in fig. 2.
In this embodiment, five acceleration cores may be configured on the FPGA on each heterogeneous acceleration card connected to the database server, but the query acceleration task of each received SQL query instruction does not need to use the five acceleration cores, but the combination of cores required for judging the query by analyzing the SQL query instruction may be one, two or three different, or five.
It should be understood that for single-period concurrent query tasks, the tasks are typically assigned to multiple boards. However, since a query task often needs to cooperate with multiple acceleration cores, different cores on a single board card are required to form a pipeline (a specific pipeline can be customized) to complete the data processing task. That is, if the query task needs to be sequenced first and then aggregated, after the sequencing is performed by the heterogeneous accelerator card 1, the sequencing result is not required to be returned to the database server, and then the sequencing result is transmitted to the aggregation kernel of the heterogeneous accelerator card 1 by the database server. The sorting kernel of the heterogeneous accelerator card 1 directly transmits the sorting result to the next aggregation kernel, then executes aggregation processing, and the server is given after all processing is completed. That is, the FPGA core on the device side adopts a modular design to achieve the maximum parallelism capability and the resource multiplexing capability, and the application layer query requirement is mapped into one or more different cores, so that the core scheduling technology accelerates the query execution through hardware core combination and pipeline optimization. The hardware core combination refers to the mapping from an application operator to a device side operator, and the hardware cores with the same computing capacity can be freely selected according to the parallel query requirement in the mapping process. Considering that most of application programs comprise tasks in parallel operation relation and tasks in sequence operation relation, kernel pipelining is the task aiming at the sequence operation relation, the copy of uplink and downlink data resources of a host and equipment is reduced, relevant bandwidth resources are saved, the intensity of calculation tasks is improved, and the running time is reduced.
Referring to fig. 4, the embodiment of the application also correspondingly discloses a multi-card collaborative database query device, which comprises:
the time estimation module 11 is used for determining the estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm when the SQL query instruction sent by the client is received through the CPU of the database server;
The accelerator card detection module 12 is configured to determine whether the estimated processing time is less than a corresponding preset time threshold, and if yes, detect, by using a preset accelerator card scheduling rule, a working state of each heterogeneous accelerator card currently connected to the database server and each accelerator core on the FPGA chip on each heterogeneous accelerator card, and determine a target heterogeneous accelerator card for processing the SQL query instruction;
the data sending module 13 is configured to send, through the CPU and the PCIE interface, to-be-processed data and objective function control parameters obtained after the SQL query instruction is parsed to the target heterogeneous accelerator card;
the data processing module 14 is configured to process the data to be processed by using the objective function control parameter through the objective heterogeneous acceleration card, so as to obtain a corresponding processing result;
the processing result obtaining module 15 is configured to obtain, through the PCIE interface corresponding to the processing result after the CPU receives the processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card, so as to complete a corresponding data query operation.
The more specific working process of each module may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
Therefore, in the embodiment of the application, when the SQL query instruction sent by the client is received through the CPU of the database server, the estimated processing time corresponding to the SQL query instruction is determined based on the preset time estimation algorithm; judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction; the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed are sent to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface; processing the data to be processed by the target heterogeneous acceleration card by utilizing the target function control parameters so as to obtain a corresponding processing result; after the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, the processing result stored in the DDR4 memory on the target heterogeneous accelerator card is acquired through the corresponding PCIE interface so as to complete corresponding data query operation. In other words, when the SQL query instruction is received, firstly, the processing time of the instruction is estimated, then, whether the estimated processing time is smaller than a preset time threshold value is judged, and when the estimated processing time is smaller than the preset time threshold value, the target heterogeneous acceleration cards for processing the instruction are distributed by detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card. And then processing the data to be processed based on the received objective function control parameters through the objective heterogeneous accelerator card, and sending a corresponding processing completion signal to the CPU after the processing is completed. Thus, the database performance can be effectively improved, and the database query efficiency is improved.
In some embodiments, the time estimation module 11 may specifically include:
The estimated processing time determining unit is used for determining the estimated processing time corresponding to the SQL query instruction based on the writing stage time overhead operator, the reading stage time overhead multiplier, the query operation stage time overhead operator and preset weight information corresponding to the client.
In some specific embodiments, the multi-card collaborative database query device may specifically further include:
and the CPU processing unit is used for processing the SQL query instruction through the CPU if the estimated processing time is not smaller than the corresponding preset time threshold value.
In some embodiments, the acceleration card detection module 12 may specifically include:
The task sending unit is used for sending task information corresponding to the SQL query instruction to a preconfigured heterogeneous task execution list if the SQL query instruction is the SQL query instruction;
The acceleration card working state detection unit is used for sequentially detecting the working states of the heterogeneous acceleration cards connected with the database server through a preset acceleration card scheduling rule;
The acceleration kernel working state detection unit is used for judging whether the working state of a target acceleration kernel corresponding to the SQL query instruction on the FPGA chip on the current heterogeneous acceleration card is in an idle state or not based on the task information when the working state of the current heterogeneous acceleration card is detected to be in a busy state, so as to obtain a corresponding judgment result;
And the acceleration card updating unit is used for updating the current heterogeneous acceleration card if the judging result is negative, and detecting the working state of the new current heterogeneous acceleration card until the target heterogeneous acceleration card for processing the SQL query instruction is determined.
In some specific embodiments, the multi-card collaborative database query device may specifically further include:
The first target heterogeneous accelerator card determines that the unit takes the current heterogeneous accelerator card as the target heterogeneous accelerator card for processing the SQL query instruction when detecting that the working state of the current heterogeneous accelerator card is an idle state;
and determining a second target heterogeneous accelerator card, wherein when the judging result is yes, the unit takes the current heterogeneous accelerator card as the target heterogeneous accelerator card for processing the SQL query instruction.
In some specific embodiments, the data sending module 13 may specifically include:
the data to be processed sending unit is used for sending the data to be processed obtained after the SQL query instruction is analyzed to the DDR4 memory of the target heterogeneous accelerator card through the CPU and the corresponding PCIE interface;
The parameter sending unit is used for sending the objective function control parameters obtained after the SQL query instruction is analyzed to the static area of the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface; wherein, the objective function control parameters comprise data bit width and total data processing amount.
In some embodiments, the data processing module 14 may specifically include:
and the data processing unit is used for transmitting the objective function control parameters stored in the static area to corresponding acceleration cores in the process of processing the data to be processed through the objective heterogeneous acceleration card, so that the acceleration cores process the data to be processed, which are acquired from the DDR4 memory through an AXI4 bus, based on the objective function control parameters.
Further, the embodiment of the present application further discloses an electronic device, and fig. 5 is a block diagram of an electronic device 20 according to an exemplary embodiment, where the content of the figure is not to be considered as any limitation on the scope of use of the present application.
Fig. 5 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, where the computer program is loaded and executed by the processor 21 to implement relevant steps in the multi-card collaborative database query method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the multi-card collaborative database query method performed by the electronic device 20 disclosed in any of the previous embodiments.
Further, the application also discloses a computer readable storage medium for storing a computer program; the computer program, when executed by the processor, realizes the multi-card collaborative database query method disclosed in the prior art. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing has outlined rather broadly the more detailed description of the application in order that the detailed description of the application that follows may be better understood, and in order that the present principles and embodiments may be better understood; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (8)
1. The multi-card collaborative database query method is characterized by comprising the following steps of:
When receiving an SQL query instruction sent by a client through a CPU of a database server, determining estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm;
judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction; the SQL queries of two layers in the equipment and among the equipment are parallel through a scheduling strategy of multi-card cooperation among the heterogeneous accelerator cards;
the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed are sent to the objective heterogeneous accelerator card through the CPU and the corresponding PCIE interface;
Processing the data to be processed by the target heterogeneous accelerator card by utilizing the target function control parameters and corresponding kernel pipelining to obtain a corresponding processing result; the kernel pipelining is a pipelining formed by different acceleration kernels, which is determined based on the processing requirements of the SQL query instructions when the execution of the query requirements of the application side is accelerated through hardware kernel combination and pipelining optimization according to a kernel scheduling technology; the hardware core combination refers to a combination freely selected by hardware cores with the same computing capacity according to parallel query requirements in the process of mapping an application operator to a device side operator;
After the CPU receives a processing completion signal sent by the target heterogeneous accelerator card, acquiring the processing result stored in the DDR4 memory on the target heterogeneous accelerator card through the corresponding PCIE interface so as to complete corresponding data query operation;
the determining the estimated processing time corresponding to the SQL query instruction based on the preset time estimation algorithm comprises the following steps:
Determining estimated processing time corresponding to the SQL query instruction based on a writing stage time overhead operator, a reading stage time overhead multiplier, a query operation stage time overhead operator and preset weight information corresponding to the client;
if yes, detecting working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on an FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction, wherein the method comprises the following steps:
If yes, task information corresponding to the SQL query instruction is sent to a preconfigured heterogeneous task execution list;
Sequentially detecting the working states of the heterogeneous accelerator cards connected with the database server through a preset accelerator card scheduling rule;
when the working state of the current heterogeneous accelerator card is detected to be a busy state, judging whether the working state of a target accelerator core corresponding to the SQL query instruction on an FPGA chip on the current heterogeneous accelerator card is an idle state or not based on the task information, and obtaining a corresponding judging result;
If the judging result is negative, updating the current heterogeneous accelerator card, and detecting the working state of the new current heterogeneous accelerator card until the target heterogeneous accelerator card for processing the SQL query instruction is determined.
2. The multi-card collaborative database query method according to claim 1, wherein after determining whether the estimated processing time is less than a corresponding predetermined time threshold, further comprising:
and if the estimated processing time is not smaller than the corresponding preset time threshold, processing the SQL query instruction through the CPU.
3. The multi-card collaborative database query method according to claim 1, further comprising:
when the working state of the current heterogeneous accelerator card is detected to be an idle state, the current heterogeneous accelerator card is used as a target heterogeneous accelerator card for processing the SQL query instruction;
And when the judgment result is yes, taking the current heterogeneous accelerator card as a target heterogeneous accelerator card for processing the SQL query instruction.
4. The multi-card collaborative database query method according to any one of claims 1 to 3, wherein the sending, through the CPU and a PCIE interface corresponding thereto, to the target heterogeneous acceleration card, data to be processed and target function control parameters obtained after parsing the SQL query instruction includes:
Sending the data to be processed obtained after analyzing the SQL query instruction to the DDR4 memory of the target heterogeneous accelerator card through the CPU and the corresponding PCIE interface;
The CPU and the corresponding PCIE interface send target function control parameters obtained after analyzing the SQL query instruction to a static area of the target heterogeneous accelerator card; wherein, the objective function control parameters comprise data bit width and total data processing amount.
5. The multi-card collaborative database query method according to claim 4, wherein the processing the data to be processed by the target heterogeneous acceleration card using the target function control parameter and the corresponding kernel pipeline to obtain a corresponding processing result comprises:
and in the process of processing the data to be processed through the target heterogeneous accelerator card, transmitting the target function control parameters stored in the static area to corresponding acceleration cores so that the acceleration cores process the data to be processed, which are acquired from the DDR4 memory through an AXI4 bus, based on the target function control parameters and corresponding core pipelining.
6. A multi-card collaborative database query apparatus, comprising:
The time estimation module is used for determining the estimated processing time corresponding to the SQL query instruction based on a preset time estimation algorithm when the SQL query instruction sent by the client is received through the CPU of the database server;
The acceleration card detection module is used for judging whether the estimated processing time is smaller than a corresponding preset time threshold value, if so, detecting the working states of each heterogeneous acceleration card currently connected with the database server and each acceleration core on the FPGA chip on each heterogeneous acceleration card through a preset acceleration card scheduling rule, and determining a target heterogeneous acceleration card for processing the SQL query instruction; the SQL queries of two layers in the equipment and among the equipment are parallel through a scheduling strategy of multi-card cooperation among the heterogeneous accelerator cards;
The data sending module is used for sending the data to be processed and the objective function control parameters obtained after the SQL query instruction is analyzed to the objective heterogeneous acceleration card through the CPU and the corresponding PCIE interface;
the data processing module is used for processing the data to be processed by the target heterogeneous acceleration card through the target function control parameters and the corresponding kernel pipelining so as to obtain a corresponding processing result; the kernel pipelining is a pipelining formed by different acceleration kernels, which is determined based on the processing requirements of the SQL query instructions when the execution of the query requirements of the application side is accelerated through hardware kernel combination and pipelining optimization according to a kernel scheduling technology; the hardware core combination refers to a combination freely selected by hardware cores with the same computing capacity according to parallel query requirements in the process of mapping an application operator to a device side operator;
the processing result acquisition module is used for acquiring the processing result stored in the DDR4 memory on the target heterogeneous accelerator card through the corresponding PCIE interface after the CPU receives the processing completion signal sent by the target heterogeneous accelerator card, so as to complete corresponding data query operation;
Wherein, the time estimation module includes:
The estimated processing time determining unit is used for determining the estimated processing time corresponding to the SQL query instruction based on the writing stage time overhead operator, the reading stage time overhead multiplier, the query operation stage time overhead operator and preset weight information corresponding to the client;
in some embodiments, the accelerator card detection module 12 includes:
The task sending unit is used for sending task information corresponding to the SQL query instruction to a preconfigured heterogeneous task execution list if the SQL query instruction is the SQL query instruction;
The acceleration card working state detection unit is used for sequentially detecting the working states of the heterogeneous acceleration cards connected with the database server through a preset acceleration card scheduling rule;
The acceleration kernel working state detection unit is used for judging whether the working state of a target acceleration kernel corresponding to the SQL query instruction on the FPGA chip on the current heterogeneous acceleration card is in an idle state or not based on the task information when the working state of the current heterogeneous acceleration card is detected to be in a busy state, so as to obtain a corresponding judgment result;
And the acceleration card updating unit is used for updating the current heterogeneous acceleration card if the judging result is negative, and detecting the working state of the new current heterogeneous acceleration card until the target heterogeneous acceleration card for processing the SQL query instruction is determined.
7. An electronic device, comprising:
a memory for storing a computer program;
A processor for executing the computer program to implement the multi-card collaborative database query method of any one of claims 1-5.
8. A computer readable storage medium for storing a computer program which when executed by a processor implements the multi-card collaborative database query method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410389664.5A CN117971906B (en) | 2024-04-02 | Multi-card collaborative database query method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410389664.5A CN117971906B (en) | 2024-04-02 | Multi-card collaborative database query method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117971906A CN117971906A (en) | 2024-05-03 |
| CN117971906B true CN117971906B (en) | 2024-07-02 |
Family
ID=
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383248A (en) * | 2023-04-28 | 2023-07-04 | 山东浪潮科学研究院有限公司 | Data query task acceleration method, device, equipment and storage medium |
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383248A (en) * | 2023-04-28 | 2023-07-04 | 山东浪潮科学研究院有限公司 | Data query task acceleration method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11010681B2 (en) | Distributed computing system, and data transmission method and apparatus in distributed computing system | |
| US10877766B2 (en) | Embedded scheduling of hardware resources for hardware acceleration | |
| US8291427B2 (en) | Scheduling applications for execution on a plurality of compute nodes of a parallel computer to manage temperature of the nodes during execution | |
| US8381230B2 (en) | Message passing with queues and channels | |
| US20080195840A1 (en) | Identifying Messaging Completion on a Parallel Computer | |
| US8898422B2 (en) | Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration | |
| US20090327444A1 (en) | Dynamic Network Link Selection For Transmitting A Message Between Compute Nodes Of A Parallel Comput | |
| CN107908536B (en) | Performance evaluation method and system for GPU application in CPU-GPU heterogeneous environment | |
| CN102340489B (en) | Data transmission method between servers and servers | |
| US11372594B2 (en) | Method and apparatus for scheduling memory access request, device and storage medium | |
| CN104182278A (en) | Method and device for judging busy degree of computer hardware resource | |
| TW202029064A (en) | Multipath neural network, method to allocate resources and multipath neural network analyzer | |
| Ubarhande et al. | Novel data-distribution technique for Hadoop in heterogeneous cloud environments | |
| CN103729252B (en) | The method and dispatching and monitoring device of a kind of scheduling virtual machine | |
| JP2014120097A (en) | Information processor, program, and information processing method | |
| CN114880346A (en) | Data processing method, related assembly and accelerated processor | |
| US20210398013A1 (en) | Method and system for performance tuning and performance tuning device | |
| US20120203763A1 (en) | Database query optimizer that takes network choice into consideration | |
| Wu et al. | Hierarchical task mapping for parallel applications on supercomputers | |
| CN109426623A (en) | A kind of method and device reading data | |
| Goh et al. | Design and performance evaluation of combined first-fit task allocation and migration strategies in mesh multiprocessor systems | |
| Chen et al. | Dosas: Mitigating the resource contention in active storage systems | |
| US11429299B2 (en) | System and method for managing conversion of low-locality data into high-locality data | |
| CN117971906B (en) | Multi-card collaborative database query method, device, equipment and storage medium | |
| CN116089477B (en) | Distributed training method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |