CN112115160A - Query request scheduling method and device and computer system - Google Patents
Query request scheduling method and device and computer system Download PDFInfo
- Publication number
- CN112115160A CN112115160A CN202010816893.2A CN202010816893A CN112115160A CN 112115160 A CN112115160 A CN 112115160A CN 202010816893 A CN202010816893 A CN 202010816893A CN 112115160 A CN112115160 A CN 112115160A
- Authority
- CN
- China
- Prior art keywords
- data
- queried
- preset
- inquired
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a scheduling method, a device and a computer system of a query request, wherein the method comprises the following steps: receiving a query request of a user; analyzing and determining data to be queried corresponding to the query request; determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried; the data to be inquired is distributed to the preset threads according to the preset distribution strategy corresponding to the data quantity to be inquired, so that the preset threads can acquire the data to be inquired and return the data to the user, the corresponding thread distribution strategy is selected according to the data quantity to be inquired corresponding to the inquiry request, the data to be inquired is distributed to one or more preset threads according to the selected strategy, the preset threads are used for acquiring corresponding data from the database so as to return the data to the user, and the problems that the other inquiry requests need to wait for a long time and the processing timeliness of the inquiry requests with small data quantity is influenced due to the fact that the data to be inquired with large data quantity is not properly distributed to the threads are avoided.
Description
Technical Field
The invention relates to the field of big data analysis, in particular to a method and a device for scheduling a query request and a computer system.
Background
With the continuous evolution and popularization of various company big data platforms, various OLAP (online analytical processing) platforms are continuously accessed to various business scenes, the query scene and complexity of an OLAP bottom layer engine are continuously promoted, and the resource distribution is unreasonable when tasting, so that the resource utilization rate is low.
Meanwhile, the query requests are also easily influenced by each other, for example, a query with a large data volume may cause a query with a small data volume to wait for a long time, and a query with a high priority may have to wait for a query with a lower priority. Therefore, a technical solution capable of reasonably scheduling the resources of the OLAP is needed.
Disclosure of Invention
In order to solve the defects of the prior art, the main object of the present invention is to provide a method, an apparatus and a computer system for scheduling a query request, so as to implement reasonable scheduling of OLAP resources.
In order to achieve the above object, a first aspect of the present invention provides a method for scheduling query requests, where the method includes:
receiving a query request of a user;
analyzing and determining the data to be queried corresponding to the query request;
determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
In some embodiments, the query request comprises an aggregated query, the method comprising:
determining a cardinal number corresponding to the data to be queried according to metadata corresponding to the data to be queried;
determining the memory capacity required by aggregating the data to be queried according to the cardinal number;
determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size;
and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
In some embodiments, the query request includes a corresponding target priority, and the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is configured to obtain the data to be queried so as to return to the user, includes:
and distributing the data to be inquired to a first preset thread in a thread pool corresponding to the target priority according to a preset distribution strategy corresponding to the quantity to be inquired, wherein the first preset thread is used for acquiring the data to be inquired so as to return the data to the user.
In some embodiments, the method comprises:
and when a first preset thread in a thread pool corresponding to the target priority is unavailable, distributing the data to be inquired to a second preset thread in the thread pool with the priority lower than the target priority, wherein the second preset thread is used for acquiring the data to be inquired so as to return the data to the user.
In some embodiments, the pre-storing the data to be queried in a data segment, where the data amount to be queried includes the number of the data segment, and the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is used to obtain the data to be queried so as to return the data to the user, includes:
and distributing the data segments to a preset thread according to a preset distribution strategy corresponding to the number of the data segments, wherein the preset thread is used for acquiring the data to be inquired from the data segments so as to return the data to the user.
In some embodiments, the query request includes an SQL statement to be queried, and the parsing and determining data to be queried corresponding to the query request includes:
and analyzing the SQL sentence to be queried, and determining a table to be queried corresponding to the query request and a column to be queried of the table to be queried.
In some embodiments, the allocating, according to a preset allocation policy corresponding to the amount of the data to be queried, the data to be queried to a preset thread, where the preset thread is configured to obtain the data to be queried so as to return to the user, includes:
when the data volume to be queried is not less than a preset threshold value, distributing the data to be queried to a preset thread according to a first preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user;
and when the data volume to be queried is smaller than a preset threshold value, distributing the data to be queried to a preset thread according to a second preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user.
In a second aspect, the present application provides an apparatus for scheduling query requests, the apparatus comprising:
the receiving module is used for receiving a query request of a user;
the analysis module is used for analyzing and determining the data to be inquired corresponding to the inquiry request;
the processing module is used for determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and the distribution module is used for distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data amount to be inquired, and the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
In some embodiments, the query request includes an aggregation query, and the apparatus further includes an aggregation module, configured to determine a cardinality corresponding to the data to be queried according to metadata corresponding to the data to be queried; determining the memory capacity required by aggregating the data to be queried according to the cardinal number; determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size; and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
In a third aspect, the present application provides a computer system comprising:
one or more processors;
and memory associated with the one or more processors for storing program instructions that, when read and executed by the one or more processors, perform operations comprising:
receiving a query request of a user;
analyzing and determining the data to be queried corresponding to the query request;
determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
The invention has the following beneficial effects:
the application provides a scheduling method of a query request, which comprises the following steps: receiving a query request of a user; analyzing and determining the data to be queried corresponding to the query request; determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried; the data to be inquired is distributed to the preset threads according to the preset distribution strategy corresponding to the data quantity to be inquired, so that the preset threads can acquire the data to be inquired and return the data to the user, the corresponding thread distribution strategy is selected according to the data quantity to be inquired corresponding to the inquiry request, the data to be inquired is distributed to one or more preset threads according to the selected strategy, the preset threads can acquire corresponding data from the database to return the data to the user, and the problems that the blockage is caused by improper distribution of the data to be inquired with large data quantity to the threads, the long-time waiting is required for other inquiry requests, and the processing timeliness of the inquiry requests with small data quantity is influenced are avoided;
further, the application provides that a cardinal number corresponding to the data to be queried is determined according to metadata corresponding to the data to be queried; determining the memory capacity required by aggregating the data to be queried according to the cardinal number; determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size; aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user, so that reasonable allocation of the memory required by aggregated query according to the data to be queried is realized, and the problems of low memory utilization rate, influence on query concurrency rate and resource waste caused by allocating too large memory blocks to the data to be queried and influence on query efficiency caused by overflowing to a disk caused by allocating too small memory blocks to the data to be queried are avoided;
the application further provides that the query request comprises a corresponding target priority, and according to the preset distribution strategy corresponding to the number to be queried, the data to be queried is distributed to a first preset thread in a thread pool corresponding to the target priority, so that the first preset thread acquires the data to be queried and returns the data to the user, and the condition that the query request with low priority occupies all threads is avoided, and the query request with high priority cannot be processed in time.
All products of the present invention need not have all of the above-described effects.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic architecture diagram of a scheduling system provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of memory block allocation provided in an embodiment of the present application;
FIG. 3 is a schematic thread pool scheduling diagram provided in an embodiment of the present application;
FIG. 4 is a flow chart of a method provided by an embodiment of the present application;
FIG. 5 is a block diagram of an apparatus according to an embodiment of the present disclosure;
fig. 6 is a computer system structure diagram provided in the embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As described in the background art, in order to solve the above problem, the present application provides a scheduling method for a query request, so as to ensure that resources of an OLAP platform can be reasonably scheduled.
Specifically, as shown in fig. 1, the present application constructs a scheduling system for query requests, and a process of scheduling query requests using the system includes:
step one, receiving a query request of a user;
a user can send a query request of data to the system through a preset query interface, wherein the query request can comprise SQL statements required to be queried by the user or data tables, columns, data rows and the like required to be queried.
Step two, analyzing the query request and determining corresponding data to be queried;
specifically, the corresponding data to be queried may be determined according to the analyzed query request. The data to be inquired comprises a data table to be inquired and a row to be inquired of the data table to be inquired.
Step three, determining the data volume to be inquired corresponding to the data to be inquired according to the metadata corresponding to the data to be inquired
The Segment is used for storing data, and the data to be queried is stored in one or more segments for storing. To determine the number of segments storing the data to be queried, metadata of the data may be pre-mined. For example, metadata of data which is less changed can be collected periodically according to a preset period; for data acquired through a data source (Datatasource), corresponding metadata can be acquired at fixed time according to preset time granularity; and collecting real-time data for the data with high real-time requirement through a real-time computing engine Flink.
For each data table, at each acquisition, the number of rows of the table, the number of segments used for storing the table, the amount of data, and the average volume of segments need to be acquired.
For each column, the number of the maximum value, the minimum value, the technology and the Null value of the column needs to be acquired during each acquisition.
According to the determined table to be queried and the column to be queried, the number of segments (data segments) storing the data to be queried can be determined.
Preferably, according to the exponential back-off algorithm, the filtering factor of the query request is further calculated to determine the number of lines to be queried by the query request.
Preferably, the cardinality of the query data may be determined based on the metadata.
Step four, distributing the query request to a thread pool corresponding to the request;
when all query requests share all threads, the query requests with low importance and large query quantity occupy resources such as threads for a long time, so that the query requests with high importance cannot preempt the resources and queue for a long time. To solve this problem, the present system presets a plurality of thread pools, each thread pool having one or more lines therein and each thread pool having a corresponding priority.
Each query request comprises a priority, the query requests can be distributed into a corresponding thread pool according to the priority of the query request, and the threads in the corresponding thread pool can be called to obtain the data to be queried from the Segment in which the data to be queried is stored.
When the thread in the thread pool corresponding to the high-priority query request is unavailable, as shown in fig. 2, the thread in the thread pool with low priority may be called by the query request with high priority, and the data to be queried is obtained from the Segment storing the data to be queried required by the query request with high priority by the thread in the thread pool with low priority.
Specifically, the unavailability of the thread includes the situations that the thread is busy and the thread has no response.
Fifthly, distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data amount to be inquired;
specifically, the allocation policy may be selected according to a preset threshold.
For example, the system employs a drive as a data store, and the drive supports two selection strategies: connection Count and Random. The Connection Count can meet the requirement of a query request with a small query quantity, but when the Connection Count is used for a request with a large query quantity, Segment distribution is often uneven, so that a certain thread or certain threads bear most of Segment query tasks, and the processing efficiency is low; while Random can meet the requirement of query requests with large query quantity, the distribution efficiency of the query requests with small query quantity is low, and the concurrency rate of the query is reduced. Therefore, in order to select an appropriate selection policy, a threshold may be preset, and when the number of segments (data segments) storing data to be queried is not less than the preset threshold, Connection Count is selected as an allocation policy, and when the number is less than the preset threshold, Random is selected as the allocation policy.
And when the allocation strategy is selected, distributing the Segment to the preset thread according to the selected allocation strategy, and acquiring the data to be inquired from the Segment storing the data to be inquired by the preset thread.
And step six, when the query request is an aggregation query, allocating a memory for the acquired data to be queried, aggregating the data to be queried in the allocated memory, and returning the aggregated data to be queried to the user.
When aggregating the results of the aggregation query, the draid needs to allocate an off-heap memory, MergeBuffer, and aggregate the data in MergeBuffer.
However, currently, the drive can only configure a fixed size, MergeBuffer. When the configured MergeBuffer has too large capacity, the memory utilization rate is not high, and the query concurrency is influenced; when the configured MergeBuffer has too small capacity, the query frequently overflows to the disk, and the query efficiency is low.
To solve this problem, as shown in fig. 3, the MergeBuffer is divided into memory blocks of different sizes in the present system.
Preferably, the size of the memory required for aggregating the data to be queried can be estimated according to the cardinality of the data to be queried according to the following formula:
e (card (x, y)), (1- ((n-1)/n) ^ p), n ═ card (x) card (y), p ═ row count, where x, y denote the column to be queried, E (card (x, y)) denotes the combined base of x, y, card (x) denotes the base of x column, and card (y) denotes the base of y column.
According to the estimated size of the memory required by the aggregation, the MergeBuffer with the smallest capacity in MergeBuffers with the capacity not smaller than the size of the required memory can be allocated as the MergeBuffer used by the aggregation operation.
For example, when 48M is estimated to be needed, 64M of the MergeBuffer may be allocated as the MergeBuffer used in the polymerization operation, and when the smallest capacity is not available, a larger memory block than the MergeBuffer with the smallest capacity is searched upwards.
The aggregated data to be queried can be returned to the user through a preset method, so that the user can analyze the data and the like.
Example two
Corresponding to the foregoing embodiment, as shown in fig. 4, the present application provides a method for scheduling an inquiry request, where the method includes:
410. receiving a query request of a user;
420. analyzing and determining the data to be queried corresponding to the query request;
430. determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
440. and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
Preferably, the query request includes a corresponding target priority, and the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is configured to obtain the data to be queried so as to return the data to be queried to the user includes:
441. and distributing the data to be inquired to a first preset thread in a thread pool corresponding to the target priority according to a preset distribution strategy corresponding to the quantity to be inquired, wherein the first preset thread is used for acquiring the data to be inquired so as to return the data to the user.
Preferably, the method comprises:
442. and when a first preset thread in a thread pool corresponding to the target priority is unavailable, distributing the data to be inquired to a second preset thread in the thread pool with the priority lower than the target priority, wherein the second preset thread is used for acquiring the data to be inquired so as to return the data to the user.
Preferably, the data to be queried is pre-stored in a data segment, the data amount to be queried includes the number of the data segment, the data to be queried is allocated to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, and the preset thread is configured to acquire the data to be queried so as to return to the user, and the method includes:
443. and distributing the data segments to a preset thread according to a preset distribution strategy corresponding to the number of the data segments, wherein the preset thread is used for acquiring the data to be inquired from the data segments so as to return the data to the user.
Preferably, the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is configured to obtain the data to be queried so as to return the data to the user includes:
444. when the data volume to be queried is not less than a preset threshold value, distributing the data to be queried to a preset thread according to a first preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user;
and when the data volume to be queried is smaller than a preset threshold value, distributing the data to be queried to a preset thread according to a second preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user.
Preferably, the query request includes an aggregated query, and the method includes:
450. determining a cardinal number corresponding to the data to be queried according to metadata corresponding to the data to be queried;
451. determining the memory capacity required by aggregating the data to be queried according to the cardinal number;
452. determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size;
453. and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
Preferably, the query request includes an SQL statement to be queried, and the analyzing and determining the data to be queried corresponding to the query request includes:
and analyzing the SQL sentence to be queried, and determining a table to be queried corresponding to the query request and a column to be queried of the table to be queried.
EXAMPLE III
Corresponding to the method, as shown in fig. 5, the present application provides a scheduling apparatus for query requests, where the apparatus includes:
a receiving module 510, configured to receive a query request of a user;
the analysis module 520 is configured to analyze and determine data to be queried corresponding to the query request;
a processing module 530, configured to determine, according to metadata corresponding to the data to be queried, a data amount to be queried corresponding to the data to be queried;
and the allocating module 540 is configured to allocate the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is configured to obtain the data to be queried so as to return the data to the user.
Preferably, the query request includes an aggregation query, and the apparatus further includes an aggregation module 550, configured to determine a cardinality corresponding to the data to be queried according to metadata corresponding to the data to be queried; determining the memory capacity required by aggregating the data to be queried according to the cardinal number; determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size; and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
Preferably, the query request includes a corresponding target priority, and the allocating module 540 is further configured to allocate the data to be queried to a first preset thread in a thread pool corresponding to the target priority according to a preset allocation policy corresponding to the quantity to be queried, where the first preset thread is used to obtain the data to be queried so as to return the data to the user.
Preferably, the allocating module 540 is further configured to, when a first preset thread in a thread pool corresponding to the target priority is unavailable, allocate the data to be queried to a second preset thread in a thread pool having a priority lower than the target priority, where the second preset thread is used to obtain the data to be queried so as to return the data to the user.
Preferably, the data to be queried is pre-stored in a data segment, the data amount to be queried includes the number of the data segment, the allocating module 540 is further configured to allocate the data segment to a preset thread according to a preset allocating policy corresponding to the number of the data segment, and the preset thread is configured to obtain the data to be queried from the data segment so as to return the data to the user.
Preferably, the query request includes an SQL statement to be queried, and the parsing module 520 may be further configured to parse the SQL statement to be queried and determine a table to be queried corresponding to the query request and a column to be queried of the table to be queried.
Preferably, the allocating module 540 is further configured to, when the amount of the data to be queried is not less than a preset threshold, allocate the data to be queried to a preset thread according to a first preset allocation policy, where the preset thread is used to obtain the data to be queried so as to return the data to the user; and when the data volume to be queried is smaller than a preset threshold value, distributing the data to be queried to a preset thread according to a second preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user.
Example four
Corresponding to the above method, apparatus, and system, a fourth embodiment of the present application provides a computer system, including: one or more processors; and memory associated with the one or more processors for storing program instructions that, when read and executed by the one or more processors, perform operations comprising:
receiving a query request of a user;
analyzing and determining the data to be queried corresponding to the query request;
determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
Fig. 6 illustrates an architecture of a computer system, which may include, in particular, a processor 1510, a video display adapter 1511, a disk drive 1512, an input/output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, video display adapter 1511, disk drive 1512, input/output interface 1513, network interface 1514, and memory 1520 may be communicatively coupled via a communication bus 1530.
The processor 1510 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solution provided by the present Application.
The Memory 1520 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1520 may store an operating system 1521 for controlling the operation of the computer system 1500, a Basic Input Output System (BIOS) for controlling low-level operations of the computer system 1500. In addition, a web browser 1523, a data storage management system 1524, an icon font processing system 1525, and the like can also be stored. The icon font processing system 1525 may be an application program that implements the operations of the foregoing steps in this embodiment of the application. In summary, when the technical solution provided by the present application is implemented by software or firmware, the relevant program codes are stored in the memory 1520 and called for execution by the processor 1510. The input/output interface 1513 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The network interface 1514 is used to connect a communication module (not shown) to enable the device to communicatively interact with other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
The bus 1530 includes a path to transfer information between the various components of the device, such as the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520.
In addition, the computer system 1500 may also obtain information of specific extraction conditions from the virtual resource object extraction condition information database 1541 for performing condition judgment, and the like.
It should be noted that although the above devices only show the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, the memory 1520, the bus 1530, etc., in a specific implementation, the devices may also include other components necessary for proper operation. Furthermore, it will be understood by those skilled in the art that the apparatus described above may also include only the components necessary to implement the solution of the present application, and not necessarily all of the components shown in the figures.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a cloud server, or a network device) to execute the method according to the embodiments or some parts of the embodiments of the present application.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for scheduling query requests, the method comprising:
receiving a query request of a user;
analyzing and determining the data to be queried corresponding to the query request;
determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
2. The method of claim 1, wherein the query request comprises an aggregated query, the method comprising:
determining a cardinal number corresponding to the data to be queried according to metadata corresponding to the data to be queried;
determining the memory capacity required by aggregating the data to be queried according to the cardinal number;
determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size;
and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
3. The method according to claim 1 or 2, wherein the query request includes a corresponding target priority, and the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, where the preset thread is configured to obtain the data to be queried so as to return to the user, includes:
and distributing the data to be inquired to a first preset thread in a thread pool corresponding to the target priority according to a preset distribution strategy corresponding to the quantity to be inquired, wherein the first preset thread is used for acquiring the data to be inquired so as to return the data to the user.
4. The method of claim 3, wherein the method comprises:
and when a first preset thread in a thread pool corresponding to the target priority is unavailable, distributing the data to be inquired to a second preset thread in the thread pool with the priority lower than the target priority, wherein the second preset thread is used for acquiring the data to be inquired so as to return the data to the user.
5. The method according to claim 1 or 2, wherein the data to be queried is pre-stored in a data segment, the data volume to be queried includes the number of the data segment, the data to be queried is allocated to a preset thread according to a preset allocation policy corresponding to the data volume to be queried, and the preset thread is used for acquiring the data to be queried so as to return the data to the user, and the method includes:
and distributing the data segments to a preset thread according to a preset distribution strategy corresponding to the number of the data segments, wherein the preset thread is used for acquiring the data to be inquired from the data segments so as to return the data to the user.
6. The method according to claim 1 or 2, wherein the query request includes an SQL statement to be queried, and the parsing and determining the data to be queried corresponding to the query request includes:
and analyzing the SQL sentence to be queried, and determining a table to be queried corresponding to the query request and a column to be queried of the table to be queried.
7. The method according to claim 1 or 2, wherein the allocating the data to be queried to a preset thread according to a preset allocation policy corresponding to the data amount to be queried, the preset thread being configured to obtain the data to be queried so as to return to the user includes:
when the data volume to be queried is not less than a preset threshold value, distributing the data to be queried to a preset thread according to a first preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user;
and when the data volume to be queried is smaller than a preset threshold value, distributing the data to be queried to a preset thread according to a second preset distribution strategy, wherein the preset thread is used for acquiring the data to be queried so as to return the data to the user.
8. An apparatus for scheduling query requests, the apparatus comprising:
the receiving module is used for receiving a query request of a user;
the analysis module is used for analyzing and determining the data to be inquired corresponding to the inquiry request;
the processing module is used for determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and the distribution module is used for distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data amount to be inquired, and the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
9. The apparatus according to claim 8, wherein the query request includes an aggregate query, and the apparatus further includes an aggregation module, configured to determine a cardinality corresponding to the data to be queried according to metadata corresponding to the data to be queried; determining the memory capacity required by aggregating the data to be queried according to the cardinal number; determining a memory block with the minimum memory capacity in the memory blocks with the memory capacity not less than the required memory capacity as a target memory block, wherein each memory block has a memory capacity with a preset size; and aggregating the acquired data to be queried in the target memory block, and returning the aggregated data to be queried to the user.
10. A computer system, the system comprising:
one or more processors;
and memory associated with the one or more processors for storing program instructions that, when read and executed by the one or more processors, perform operations comprising:
receiving a query request of a user;
analyzing and determining the data to be queried corresponding to the query request;
determining the data volume to be queried corresponding to the data to be queried according to the metadata corresponding to the data to be queried;
and distributing the data to be inquired to a preset thread according to a preset distribution strategy corresponding to the data volume to be inquired, wherein the preset thread is used for acquiring the data to be inquired so as to return the data to the user.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010816893.2A CN112115160B (en) | 2020-08-14 | 2020-08-14 | Query request scheduling method and device and computer system |
| CA3127885A CA3127885C (en) | 2020-08-14 | 2021-08-12 | Scheduling method and device for query request and computer system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010816893.2A CN112115160B (en) | 2020-08-14 | 2020-08-14 | Query request scheduling method and device and computer system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112115160A true CN112115160A (en) | 2020-12-22 |
| CN112115160B CN112115160B (en) | 2023-03-17 |
Family
ID=73804132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010816893.2A Active CN112115160B (en) | 2020-08-14 | 2020-08-14 | Query request scheduling method and device and computer system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112115160B (en) |
| CA (1) | CA3127885C (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495923A (en) * | 2021-02-09 | 2021-10-12 | 深圳市云网万店科技有限公司 | Scheduling management method and system for distributed database executor |
| CN113536075A (en) * | 2021-07-20 | 2021-10-22 | 锐掣(杭州)科技有限公司 | Data extraction method, device and storage medium |
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090077054A1 (en) * | 2007-09-13 | 2009-03-19 | Brian Robert Muras | Cardinality Statistic for Optimizing Database Queries with Aggregation Functions |
| CN110222072A (en) * | 2019-06-06 | 2019-09-10 | 江苏满运软件科技有限公司 | Data Query Platform, method, equipment and storage medium |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
-
2020
- 2020-08-14 CN CN202010816893.2A patent/CN112115160B/en active Active
-
2021
- 2021-08-12 CA CA3127885A patent/CA3127885C/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090077054A1 (en) * | 2007-09-13 | 2009-03-19 | Brian Robert Muras | Cardinality Statistic for Optimizing Database Queries with Aggregation Functions |
| CN110222072A (en) * | 2019-06-06 | 2019-09-10 | 江苏满运软件科技有限公司 | Data Query Platform, method, equipment and storage medium |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495923A (en) * | 2021-02-09 | 2021-10-12 | 深圳市云网万店科技有限公司 | Scheduling management method and system for distributed database executor |
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113609168B (en) * | 2021-07-14 | 2024-01-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113536075A (en) * | 2021-07-20 | 2021-10-22 | 锐掣(杭州)科技有限公司 | Data extraction method, device and storage medium |
| CN113536075B (en) * | 2021-07-20 | 2024-06-04 | 锐掣(杭州)科技有限公司 | Data extraction method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CA3127885A1 (en) | 2022-02-14 |
| CA3127885C (en) | 2024-04-23 |
| CN112115160B (en) | 2023-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112115160B (en) | Query request scheduling method and device and computer system | |
| US11734271B2 (en) | Data query method, apparatus and device | |
| CN109783237B (en) | Resource allocation method and device | |
| JP7511477B2 (en) | DATA QUERY METHOD, APPARATUS, AND DEVICE - Patent application | |
| US9483319B2 (en) | Job scheduling apparatus and method therefor | |
| CA3177212A1 (en) | Resource allocating method, device, computer equipment, and storage medium | |
| US9992301B2 (en) | Prioritized throttling in a multiusage environment | |
| CN108595254B (en) | Query scheduling method | |
| CN112181613B (en) | Heterogeneous resource distributed computing platform batch task scheduling method and storage medium | |
| CN110807145A (en) | Query engine acquisition method, device and computer-readable storage medium | |
| CN113867953A (en) | Resource allocation method, device, storage medium and electronic equipment | |
| CN103729417A (en) | Method and device for data scanning | |
| CN111813541B (en) | Task scheduling method, device, medium and equipment | |
| CN111427887A (en) | Method, device and system for rapidly scanning HBase partition table | |
| CN114860449B (en) | Data processing method, device, equipment and storage medium | |
| CN113360481B (en) | Data processing method, device, equipment and computer readable storage medium | |
| CN116010447A (en) | Load balancing method and device for optimizing heterogeneous database user query | |
| CN115016890A (en) | Virtual machine resource allocation method and device, electronic equipment and storage medium | |
| CN108572871B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN112527482A (en) | Task management method and system based on mobile edge cloud platform | |
| CN112433838A (en) | Batch scheduling method, device, equipment and computer storage medium | |
| CN115391042B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN112540843A (en) | Resource allocation method and device, storage equipment and storage medium | |
| CN111427682A (en) | Task allocation method, system, device and equipment | |
| CN112905351B (en) | GPU and CPU load scheduling method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |