CN114880368A - Data query method and device, electronic equipment and readable storage medium - Google Patents
Data query method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114880368A CN114880368A CN202210589925.9A CN202210589925A CN114880368A CN 114880368 A CN114880368 A CN 114880368A CN 202210589925 A CN202210589925 A CN 202210589925A CN 114880368 A CN114880368 A CN 114880368A
- Authority
- CN
- China
- Prior art keywords
- query
- data
- service
- query request
- requests
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 238000013467 fragmentation Methods 0.000 claims abstract description 92
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 92
- 239000012634 fragment Substances 0.000 claims abstract description 89
- 238000003672 processing method Methods 0.000 claims abstract description 28
- 230000004931 aggregating effect Effects 0.000 claims abstract description 17
- 230000006870 function Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 10
- 238000009826 distribution Methods 0.000 claims description 4
- 238000013439 planning Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 11
- 238000013473 artificial intelligence Methods 0.000 abstract description 7
- 238000004891 communication Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 239000004973 liquid crystal related substance Substances 0.000 description 4
- 238000013500 data storage Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to an artificial intelligence technology, and discloses a data query method, which comprises the following steps: classifying the query requests in the data query request set according to the query types, performing data fragmentation on the time query request set to obtain a query fragment set, performing task scheduling on the service query request set to obtain a service query queue, performing query processing on the fragment requests in the query fragment set by using a preset multithreading processing method, aggregating the queried fragment data to obtain a time query result, performing query processing on the requests in the service query queue by using the multithreading processing method, and taking the queried data as the service query result. In addition, the invention also relates to a block chain technology, and the time query result can be stored in a node of the block chain. The invention also provides a data query device, an electronic device and a readable storage medium. The invention can solve the problem of low data query efficiency.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a data query method and device, electronic equipment and a readable storage medium.
Background
With the popularization of the internet in the whole society, the human society begins to enter the data era, the dependence of various industries on data is higher and higher, and the production and processing requirements of the data are continuously increased. The application programs of the internet companies need to query, analyze and process the data so as to meet the diversified demands of users.
In daily work, people often encounter methods which need to query a large amount of data, such as needing to query subway riding records in the last month of a certain city, online shopping order records in the last year of a certain e-commerce platform and the like. In such a scenario of querying data within a certain time range, the size of the data has a great influence on the efficiency of the query, and the scenario of querying data with hundreds of millions or even hundreds of millions is becoming more and more common in work. However, in the current common data storage tool (database, etc.), it usually takes several minutes or even ten and several minutes to look for such a large amount of data query, and in this period of time, resources (CPU, memory, etc.) of the entire database machine are consumed by the query, and the query to other interfaces cannot be responded in time, and the entire machine is even crashed seriously, thereby resulting in low data query efficiency.
Disclosure of Invention
The invention provides a data query method, a data query device, electronic equipment and a readable storage medium, and mainly aims to solve the problem of low data query efficiency.
In order to achieve the above object, the present invention provides a data query method, including:
acquiring a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets;
performing data fragmentation on the time class query request sets in the plurality of classification query request sets to obtain a query fragmentation set;
task scheduling is carried out on the service type query request sets in the plurality of classified query request sets to obtain service query queues;
utilizing a preset multithread processing method to execute query processing on the fragment requests in the query fragment set, and aggregating the queried fragment data to obtain a time query result;
and executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
Optionally, the classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets includes:
analyzing the query request in the data query request set to obtain a request service identifier;
taking the corresponding data query request including the time range identifier in the request service identifier as a time query request, and summarizing all the time query requests to obtain a time query request set;
and classifying the corresponding data query requests which do not include the time range identifier in the request service identifier according to a preset service classification table, taking the data query requests as service query requests, and summarizing all the service query requests to obtain a service query request set.
Optionally, the performing data fragmentation on the time-class query request set in the multiple classification query request sets to obtain a query fragmentation set includes:
determining the starting time and the ending time of the time range identifier in the time class query request set;
and matching a preset fragmentation dimension table according to the time range in which the starting time and the ending time fall, and fragmenting the query requests in the time class query request set according to the matched fragmentation dimension to obtain the query fragmentation set.
Optionally, the fragmenting, according to the matched fragmentation dimension, the query request in the time-class query request set to obtain the query fragmentation set includes:
based on the fragmentation dimension, performing range fragmentation on the query request in the time class query request set by using a preset fragmentation service center to obtain an original fragmentation set;
and marking the fragment requests under the same query request in the original fragment set, and summarizing all marked fragment requests to obtain the query fragment set.
Optionally, the performing, by using a preset multithread processing method, query processing on the fragment request in the query fragment set, and aggregating the queried fragment data to obtain a time query result includes:
performing thread distribution on the fragment requests in the query fragment set according to a preset thread pool;
executing data query in a preset query database by using the distributed threads to obtain fragment data queried by each fragment request;
and aggregating the fragment data by using the mark in the fragment request to obtain the time query result.
Optionally, the task scheduling the service class query request set in the multiple classification query request sets to obtain a service query queue includes:
performing linear programming on the query request tasks in the service query request set by using a preset objective function and constraint conditions, and scheduling the tasks in the service query request set according to the size of a programming result;
the preset objective function and the constraint condition are as follows:
s.t.max(x i )≤L
wherein, P j Importance of the jth task, Y j The resource occupied by the jth task, n is the number of the service type query request set, and z isThe planning result, x i And L is the maximum number of the tasks which can be operated in the period.
Optionally, before the executing the data query in the preset query database by using the distributed threads, the method further includes:
dividing a preset database into a plurality of data sub-databases according to the service types of data stored in the preset database;
and horizontally dividing the data tables in the preset database into a plurality of data sub-tables, distributing the data sub-tables to the data sub-databases according to preset service rules, and summarizing all the data sub-databases and the data sub-tables to obtain the query database.
In order to solve the above problem, the present invention also provides a data query apparatus, including:
the query request classification module is used for acquiring a data query request set, classifying the query requests in the data query request set according to the query types of the query requests in the data query request set, and acquiring a plurality of classified query request sets;
the data fragmentation module is used for performing data fragmentation on the time class query request set in the plurality of classified query request sets to obtain a query fragmentation set;
the task scheduling module is used for performing task scheduling on the service type query request set in the plurality of classified query request sets to obtain a service query queue;
and the data query module is used for executing query processing on the fragment requests in the query fragment set by using a preset multithread processing method, aggregating the queried fragment data to obtain a time query result, executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one computer program; and
and the processor executes the computer program stored in the memory to realize the data query method.
In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the data query method described above.
The invention classifies the query requests in the data query request set according to the query type of the query request to obtain a plurality of classified query request sets, and performs data fragmentation on the time query request set to obtain the query fragment set, thereby dividing the large data quantity query task (namely the time query request) into a plurality of small tasks, greatly reducing the waiting time for executing a single large task and improving the query efficiency. And task scheduling is carried out on the service type query request set to obtain a service query queue, and the query efficiency of non-large data query tasks (namely service type query requests) is improved. Therefore, the data query method, the data query device, the electronic equipment and the computer readable storage medium provided by the invention can solve the problem of low data query efficiency.
Drawings
Fig. 1 is a schematic flow chart of a data query method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart showing a detailed implementation of one of the steps in FIG. 1;
FIG. 3 is a schematic flow chart showing another step of FIG. 1;
FIG. 4 is a schematic flow chart showing another step of FIG. 1;
FIG. 5 is a functional block diagram of a data query device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device for implementing the data query method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention provides a data query method. The execution subject of the data query method includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present invention. In other words, the data query method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Fig. 1 is a schematic flow chart of a data query method according to an embodiment of the present invention.
In this embodiment, the data query method includes the following steps S1-S5:
s1, obtaining a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets.
In the embodiment of the present invention, the data query request set refers to requests for querying data in different services, and includes a large amount of data query requests and a non-large amount of data query requests. For example, in the e-commerce platform, the massive data query request can be an online shopping order record query in the last year, and the non-massive data query can be an online shopping order record query.
In detail, referring to fig. 2, the classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets includes the following steps S10-S12:
s10, analyzing the query request in the data query request set to obtain a request service identifier;
s11, taking the corresponding data query request including the time range identifier in the request service identifier as a time query request, and summarizing all the time query requests to obtain a time query request set;
s12, classifying the corresponding data query requests which do not include the time range identifier in the request service identifier according to a preset service classification table, using the data query requests as service query requests, and summarizing all the service query requests to obtain a service query request set.
In the embodiment of the present invention, the service request identifier includes identifiers such as "order," "user name," "user," and the like, and the time range identifier refers to an identifier of a queried time range, for example, "startDate": "2015-01", "endDate": "2015-12". The preset service classification table comprises an identifier and a corresponding service. Because a large amount of data query can be caused by querying data within a certain time range, higher resources are needed to be spent, and the query efficiency is lower, so that the corresponding data query requests are classified, and the data query efficiency can be improved.
S2, performing data fragmentation on the time class query request sets in the multiple classification query request sets to obtain a query fragmentation set.
In the embodiment of the invention, because the time-class query request needs to query a large amount of data, in order to improve the efficiency of data query, the query time range is further refined by data fragmentation.
Specifically, referring to fig. 3, the data slicing for the time-class query request set in the multiple classified query request sets to obtain a query slice set includes the following steps S20-S21:
s20, determining the starting time and the ending time of the time range identifier in the time type query request set;
and S21, matching a preset fragmentation dimension table according to the time range in which the start time and the end time fall, and fragmenting the query requests in the time class query request set according to the matched fragmentation dimension to obtain the query fragmentation set.
In an optional embodiment of the present invention, the preset fragmentation dimension table is a fragmentation time dimension table comprehensively considered in advance according to factors such as the size of the fragmentation data volume, the number of fragments, and the duration of fragment query, for example, the start time and the end time are "start date": "2015-01-01", "endDate": "2015-01-15", the corresponding slicing dimension is days, i.e., the query time range is sliced on a daily basis.
In detail, the fragmenting the query request in the time-class query request set according to the matched fragmentation dimension to obtain the query fragmentation set includes:
based on the fragmentation dimension, performing range fragmentation on the query request in the time class query request set by using a preset fragmentation service center to obtain an original fragmentation set;
and marking the fragment requests under the same query request in the original fragment set, and summarizing all marked fragment requests to obtain the query fragment set.
In the embodiment of the present invention, the range fragmentation refers to dividing data into a plurality of data chunks (chunks) according to values of fields, for example, performing fragmentation according to a time dimension X, where a value range of X is [ minKey, maxKey ], dividing a whole value range into a plurality of chunks, for example, obtaining fragmentation requests of a plurality of time ranges [2015-01-01-00:00,2015-01-02-00:00], [2015-01-02-00:00,2015-01-03-00:00] and the like through range fragmentation. The preset fragmentation service center can self-define and adjust fragmentation dimension and perform fragmentation.
According to the method, the large data query task is divided into a plurality of small tasks, so that the waiting time for executing a single large task is greatly reduced, and the situations that a machine CPU is high and a database is dragged and crashed due to the fact that the single large task is executed are avoided. And the size of the fragment can be flexibly adjusted, the requirements of different scenes are met, and the condition that the large-data-volume task query fails is avoided.
S3, performing task scheduling on the service type query request set in the plurality of classified query request sets to obtain a service query queue.
In the embodiment of the invention, because different services have different importance, different occupied resources and the like, the task scheduling needs to be carried out on the query request, and the accuracy of data query is improved.
In detail, the task scheduling the service class query request set in the plurality of classified query request sets to obtain a service query queue includes:
performing linear programming on the query request tasks in the service query request set by using a preset objective function and constraint conditions, and scheduling the tasks in the service query request set according to the size of a programming result;
the preset objective function and the constraint condition are as follows:
s.t.max(x i )≤L
wherein, P j Importance of the jth task, Y j The resource occupied by the jth task, n is the number of the service type query request set, z is the planning result, and x i And L is the maximum number of the tasks which can be operated in the period.
In the embodiment of the present invention, Linear Programming (abbreviated as LP) refers to a method for solving a maximum value or a minimum value of a Linear objective function under a constraint condition of a Linear equation or an inequality. The operation efficiency condition of resources in each time period can be calculated in advance through linear programming, and the aims of ensuring the priority operation of key query tasks and realizing reasonable scheduling are fulfilled.
S4, executing query processing on the fragment requests in the query fragment set by using a preset multithread processing method, and aggregating the queried fragment data to obtain a time query result.
In the embodiment of the present invention, the preset multithreading processing method may be JAVA multithreading processing or the like. Meanwhile, the slicing tasks are processed in a multi-thread mode, and the processing efficiency of the whole process is improved.
Further, referring to fig. 4, the performing query processing on the shard requests in the query shard set by using a preset multithread processing method, and aggregating the queried shard data to obtain a time query result includes the following steps S40-S42:
s40, performing thread distribution on the fragment requests in the query fragment set according to a preset thread pool;
s41, executing data query in a preset query database by using the distributed threads to obtain the fragment data queried by each fragment request;
and S42, aggregating the fragment data by using the marks in the fragment request to obtain the time query result.
In an optional embodiment of the invention, the preset thread pool is a container for accommodating a plurality of threads, and the threads in the pool can be repeatedly used, so that the operation of frequently creating thread objects is omitted, and a large amount of time and resources are saved. Firstly, judging whether an idle thread exists in a thread pool, if so, using the thread pool, if not, creating the thread and putting the thread into the thread pool, then using the thread, and if not, waiting for the existence of the idle thread in the thread pool.
In another optional embodiment of the present invention, before the executing the data query in the preset query database by using the distributed thread, the method further includes:
dividing a preset database into a plurality of data sub-databases according to the service types of data stored in the preset database;
and horizontally dividing the data tables in the preset database into a plurality of data sub-tables, distributing the data sub-tables to the data sub-databases according to preset service rules, and summarizing all the data sub-databases and the data sub-tables to obtain the query database.
In an optional embodiment of the present invention, the service type corresponds to a service in the request service identifier, where the horizontal splitting refers to dispersing data in the same table into multiple data sub-bases or multiple data sub-tables according to a logical relationship inherent in data in the table, and each data sub-table only contains a part of data, so that the data size of a single data sub-table is reduced, a distributed effect is achieved, and query efficiency is improved.
S5, executing query processing on the request in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
In the embodiment of the invention, because the requests in the service query queue do not pass through the data fragmentation, the requests in the service query queue are queried in the query database directly by a multithreading processing method, and the obtained query data is the service query result.
The invention classifies the query requests in the data query request set according to the query type of the query request to obtain a plurality of classified query request sets, and performs data fragmentation on the time query request set to obtain the query fragment set, thereby dividing the large data quantity query task (namely the time query request) into a plurality of small tasks, greatly reducing the waiting time for executing a single large task and improving the query efficiency. And task scheduling is carried out on the service type query request set to obtain a service query queue, and the query efficiency of non-large data query tasks (namely service type query requests) is improved. Therefore, the data query method provided by the invention can solve the problem of low data query efficiency.
Fig. 5 is a functional block diagram of a data query apparatus according to an embodiment of the present invention.
The data query apparatus 100 according to the present invention may be installed in an electronic device. According to the implemented functions, the data query apparatus 100 may include a query request classification module 101, a data fragmentation module 102, a task scheduling module 103, and a data query module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the query request classification module 101 is configured to obtain a data query request set, classify query requests in the data query request set according to query types of the query requests in the data query request set, and obtain a plurality of classification query request sets;
the data fragmentation module 102 is configured to perform data fragmentation on a time class query request set in the multiple classification query request sets to obtain a query fragmentation set;
the task scheduling module 103 is configured to perform task scheduling on a service class query request set in the multiple classification query request sets to obtain a service query queue;
the data query module 104 is configured to perform query processing on the fragment requests in the query fragment set by using a preset multithread processing method, aggregate the queried fragment data to obtain a time query result, perform query processing on the requests in the service query queue by using the multithread processing method, and use the queried data as a service query result.
In detail, the specific implementation of each module of the data query apparatus 100 is as follows:
step one, acquiring a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets.
In the embodiment of the present invention, the data query request set refers to requests for querying data in different services, and includes a large amount of data query requests and a non-large amount of data query requests. For example, in the e-commerce platform, the massive data query request can be an online shopping order record query in the last year, and the non-massive data query can be an online shopping order record query.
In detail, the classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets includes:
analyzing the query request in the data query request set to obtain a request service identifier;
taking the corresponding data query request including the time range identifier in the request service identifier as a time query request, and summarizing all the time query requests to obtain a time query request set;
and classifying the corresponding data query requests which do not include the time range identifier in the request service identifier according to a preset service classification table, taking the data query requests as service query requests, and summarizing all the service query requests to obtain a service query request set.
In the embodiment of the present invention, the service request identifier includes identifiers such as "order," "user name," "user," and the like, and the time range identifier refers to an identifier of a queried time range, for example, "startDate": "2015-01", "endDate": "2015-12". The preset service classification table comprises an identifier and a corresponding service. Because a large amount of data query can be caused by querying data within a certain time range, higher resources are needed to be spent, and the query efficiency is lower, so that the corresponding data query requests are classified, and the data query efficiency can be improved.
And step two, performing data fragmentation on the time class query request sets in the plurality of classification query request sets to obtain a query fragmentation set.
In the embodiment of the invention, because the time-class query request needs to query a large amount of data, in order to improve the efficiency of data query, the query time range is further refined by data fragmentation.
Specifically, the performing data fragmentation on the time-class query request set in the multiple classified query request sets to obtain a query fragmentation set includes:
determining the starting time and the ending time of the time range identifier in the time class query request set;
and matching a preset fragmentation dimension table according to the time range in which the starting time and the ending time fall, and fragmenting the query requests in the time class query request set according to the matched fragmentation dimension to obtain the query fragmentation set.
In an optional embodiment of the present invention, the preset fragmentation dimension table is a fragmentation time dimension table comprehensively considered in advance according to factors such as the size of the fragmentation data volume, the number of fragments, and the duration of fragment query, for example, the start time and the end time are "start date": "2015-01-01", "endDate": "2015-01-15", the corresponding slicing dimension is days, i.e., the query time range is sliced on a daily basis.
In detail, the fragmenting the query request in the time-class query request set according to the matched fragmentation dimension to obtain the query fragmentation set includes:
based on the fragmentation dimension, performing range fragmentation on the query request in the time class query request set by using a preset fragmentation service center to obtain an original fragmentation set;
and marking the fragment requests under the same query request in the original fragment set, and summarizing all marked fragment requests to obtain the query fragment set.
In the embodiment of the present invention, the range fragmentation refers to dividing data into a plurality of data chunks (chunks) according to values of fields, for example, performing fragmentation according to a time dimension X, where a value range of X is [ minKey, maxKey ], dividing a whole value range into a plurality of chunks, for example, obtaining fragmentation requests of a plurality of time ranges [2015-01-01-00:00,2015-01-02-00:00], [2015-01-02-00:00,2015-01-03-00:00] and the like through range fragmentation. The preset fragmentation service center can self-define and adjust fragmentation dimension and perform fragmentation.
The invention divides the large data query task into a plurality of small tasks, thereby greatly reducing the waiting time for executing a single large task, and simultaneously avoiding the situations of high CPU drag and collapse of the database caused by executing the single large task. And the size of the fragment can be flexibly adjusted, the requirements of different scenes are met, and the condition that the large-data-volume task query fails is avoided.
And step three, performing task scheduling on the service type query request sets in the plurality of classified query request sets to obtain a service query queue.
In the embodiment of the invention, because different services have different importance, different occupied resources and the like, the task scheduling needs to be carried out on the query request, and the accuracy of data query is improved.
In detail, the task scheduling the service class query request set in the plurality of classified query request sets to obtain a service query queue includes:
performing linear programming on the query request tasks in the service query request set by using a preset objective function and constraint conditions, and scheduling the tasks in the service query request set according to the size of a programming result;
the preset objective function and the constraint condition are as follows:
s.t.max(x i )≤L
wherein, P j Importance of the jth task, Y j The resource occupied by the jth task, n is the number of the service type query request set, z is the planning result, and x i And L is the maximum number of the tasks which can be operated in the period.
In the embodiment of the present invention, Linear Programming (abbreviated as LP) refers to a method for solving a maximum value or a minimum value of a Linear objective function under a constraint condition of a Linear equation or an inequality. The operation efficiency condition of resources in each time period can be calculated in advance through linear programming, and the aims of ensuring the priority operation of key query tasks and realizing reasonable scheduling are fulfilled.
And fourthly, executing query processing on the fragment requests in the query fragment set by using a preset multithread processing method, and aggregating the queried fragment data to obtain a time query result.
In the embodiment of the present invention, the preset multithreading processing method may be JAVA multithreading processing or the like. Meanwhile, the slicing tasks are processed in a multi-thread mode, and the processing efficiency of the whole process is improved.
Further, the performing query processing on the fragment requests in the query fragment set by using a preset multithread processing method, and aggregating the queried fragment data to obtain a time query result includes:
performing thread distribution on the fragment requests in the query fragment set according to a preset thread pool;
executing data query in a preset query database by using the distributed threads to obtain fragment data queried by each fragment request;
and aggregating the fragment data by using the mark in the fragment request to obtain the time query result.
In an optional embodiment of the invention, the preset thread pool is a container for accommodating a plurality of threads, and the threads in the pool can be repeatedly used, so that the operation of frequently creating thread objects is omitted, and a large amount of time and resources are saved. Firstly, judging whether an idle thread exists in a thread pool, if so, using the thread pool, if not, creating the thread and putting the thread into the thread pool, then using the thread, and if not, waiting for the existence of the idle thread in the thread pool.
In another optional embodiment of the present invention, before the executing the data query in the preset query database by using the distributed threads, the method further includes:
dividing a preset database into a plurality of data sub-databases according to the service types of data stored in the preset database;
and horizontally dividing the data tables in the preset database into a plurality of data sub-tables, distributing the data sub-tables to the data sub-databases according to preset service rules, and summarizing all the data sub-databases and the data sub-tables to obtain the query database.
In an optional embodiment of the present invention, the service type corresponds to a service in the request service identifier, where the horizontal splitting refers to dispersing data in the same table into multiple data sub-bases or multiple data sub-tables according to a logical relationship inherent in data in the table, and each data sub-table only contains a part of data, so that the data size of a single data sub-table is reduced, a distributed effect is achieved, and query efficiency is improved.
And fifthly, executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
In the embodiment of the invention, because the requests in the service query queue do not pass through the data fragmentation, the requests in the service query queue are queried in the query database directly by a multithreading processing method, and the obtained query data is the service query result.
The invention classifies the query requests in the data query request set according to the query type of the query request to obtain a plurality of classified query request sets, and performs data fragmentation on the time query request set to obtain the query fragment set, thereby dividing the large data quantity query task (namely the time query request) into a plurality of small tasks, greatly reducing the waiting time for executing a single large task and improving the query efficiency. And task scheduling is carried out on the service type query request set to obtain a service query queue, and the query efficiency of non-large data query tasks (namely service type query requests) is improved. Therefore, the data query device provided by the invention can solve the problem of low data query efficiency.
Fig. 6 is a schematic structural diagram of an electronic device implementing a data query method according to an embodiment of the present invention.
The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may further include a computer program, such as a data query program, stored in the memory 11 and operable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 11 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used not only to store application software installed in the electronic device and various types of data, such as codes of a data query program, etc., but also to temporarily store data that has been output or will be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (e.g., data query programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The communication interface 12 is used for communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which are commonly used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
The bus 13 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 13 may be divided into an address bus, a data bus, a control bus, etc. The bus 13 is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 6 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 6 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device and other electronic devices.
Optionally, the electronic device may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data query program stored in the memory 11 of the electronic device is a combination of instructions, and when executed in the processor 10, can realize:
acquiring a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets;
performing data fragmentation on the time class query request sets in the plurality of classification query request sets to obtain a query fragmentation set;
task scheduling is carried out on the service type query request sets in the plurality of classified query request sets to obtain service query queues;
utilizing a preset multithread processing method to execute query processing on the fragment requests in the query fragment set, and aggregating the queried fragment data to obtain a time query result;
and executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
Specifically, the specific implementation method of the instruction by the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to the drawings, which is not described herein again.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
acquiring a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets;
performing data fragmentation on the time class query request sets in the plurality of classification query request sets to obtain a query fragmentation set;
task scheduling is carried out on the service type query request sets in the plurality of classified query request sets to obtain service query queues;
utilizing a preset multithread processing method to execute query processing on the fragment requests in the query fragment set, and aggregating the queried fragment data to obtain a time query result;
and executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The embodiment of the invention can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not to denote any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method for data query, the method comprising:
acquiring a data query request set, and classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets;
performing data fragmentation on the time class query request sets in the plurality of classification query request sets to obtain a query fragmentation set;
task scheduling is carried out on the service type query request sets in the plurality of classified query request sets to obtain service query queues;
utilizing a preset multithread processing method to execute query processing on the fragment requests in the query fragment set, and aggregating the queried fragment data to obtain a time query result;
and executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
2. The data query method of claim 1, wherein the classifying the query requests in the data query request set according to the query types of the query requests in the data query request set to obtain a plurality of classified query request sets comprises:
analyzing the query request in the data query request set to obtain a request service identifier;
taking the corresponding data query request including the time range identifier in the request service identifier as a time query request, and summarizing all the time query requests to obtain a time query request set;
and classifying the corresponding data query requests which do not include the time range identifier in the request service identifier according to a preset service classification table, taking the data query requests as service query requests, and summarizing all the service query requests to obtain a service query request set.
3. The data query method of claim 2, wherein the performing data fragmentation on the time class query request set of the plurality of classified query request sets to obtain a query fragmentation set comprises:
determining the starting time and the ending time of the time range identifier in the time class query request set;
and matching a preset fragmentation dimension table according to the time range in which the starting time and the ending time fall, and fragmenting the query requests in the time class query request set according to the matched fragmentation dimension to obtain the query fragmentation set.
4. The data query method of claim 3, wherein the fragmenting the query request in the time-class query request set according to the matched fragmentation dimension to obtain the query fragmentation set comprises:
based on the fragmentation dimension, performing range fragmentation on the query request in the time class query request set by using a preset fragmentation service center to obtain an original fragmentation set;
and marking the fragment requests under the same query request in the original fragment set, and summarizing all marked fragment requests to obtain the query fragment set.
5. The data query method of claim 1, wherein the performing query processing on the fragment requests in the query fragment set by using a preset multithread processing method and aggregating the queried fragment data to obtain a time query result comprises:
performing thread distribution on the fragment requests in the query fragment set according to a preset thread pool;
executing data query in a preset query database by using the distributed threads to obtain fragment data queried by each fragment request;
and aggregating the fragment data by using the mark in the fragment request to obtain the time query result.
6. The data query method of claim 1, wherein task scheduling the service class query request set of the plurality of classified query request sets to obtain a service query queue comprises:
performing linear programming on the query request tasks in the service query request set by using a preset objective function and constraint conditions, and scheduling the tasks in the service query request set according to the size of a programming result;
the preset objective function and the constraint condition are as follows:
s.t.max(x i )≤L
wherein, P j Is the importance of the jth task, Y j The resource occupied by the jth task, n is the number of the service type query request set, z is the planning result, and x i And L is the maximum number of the tasks which can be operated in the period.
7. The data query method of claim 5, wherein before performing a data query in a preset query database using a distributed thread, the method further comprises:
dividing a preset database into a plurality of data sub-databases according to the service types of data stored in the preset database;
and horizontally dividing the data tables in the preset database into a plurality of data sub-tables, distributing the data sub-tables to the data sub-databases according to preset service rules, and summarizing all the data sub-databases and the data sub-tables to obtain the query database.
8. A data query apparatus, characterized in that the apparatus comprises:
the query request classification module is used for acquiring a data query request set, classifying the query requests in the data query request set according to the query types of the query requests in the data query request set, and acquiring a plurality of classified query request sets;
the data fragmentation module is used for performing data fragmentation on the time class query request set in the plurality of classified query request sets to obtain a query fragmentation set;
the task scheduling module is used for performing task scheduling on the service type query request set in the plurality of classified query request sets to obtain a service query queue;
and the data query module is used for executing query processing on the fragment requests in the query fragment set by using a preset multithread processing method, aggregating the queried fragment data to obtain a time query result, executing query processing on the requests in the service query queue by using the multithread processing method, and taking the queried data as a service query result.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data query method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out a data query method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589925.9A CN114880368A (en) | 2022-05-26 | 2022-05-26 | Data query method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210589925.9A CN114880368A (en) | 2022-05-26 | 2022-05-26 | Data query method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114880368A true CN114880368A (en) | 2022-08-09 |
Family
ID=82678538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210589925.9A Pending CN114880368A (en) | 2022-05-26 | 2022-05-26 | Data query method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114880368A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115952202A (en) * | 2023-02-24 | 2023-04-11 | 北京锐服信科技有限公司 | Query method |
| CN117349323A (en) * | 2023-12-05 | 2024-01-05 | 苏州元脑智能科技有限公司 | Database data processing method and device, storage medium and electronic equipment |
-
2022
- 2022-05-26 CN CN202210589925.9A patent/CN114880368A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115952202A (en) * | 2023-02-24 | 2023-04-11 | 北京锐服信科技有限公司 | Query method |
| CN115952202B (en) * | 2023-02-24 | 2023-08-22 | 北京锐服信科技有限公司 | Big data fragment query method based on cache |
| CN117349323A (en) * | 2023-12-05 | 2024-01-05 | 苏州元脑智能科技有限公司 | Database data processing method and device, storage medium and electronic equipment |
| CN117349323B (en) * | 2023-12-05 | 2024-02-27 | 苏州元脑智能科技有限公司 | Database data processing method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| CN112115152B (en) | Data increment updating and inquiring method and device, electronic equipment and storage medium | |
| CN112541745A (en) | User behavior data analysis method and device, electronic equipment and readable storage medium | |
| CN114880368A (en) | Data query method and device, electronic equipment and readable storage medium | |
| CN112418798A (en) | Information auditing method and device, electronic equipment and storage medium | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112231417A (en) | Data classification method and device, electronic equipment and storage medium | |
| CN113868528A (en) | Information recommendation method and device, electronic equipment and readable storage medium | |
| CN114881616A (en) | Business process execution method and device, electronic equipment and storage medium | |
| CN112256783A (en) | Data export method and device, electronic equipment and storage medium | |
| CN114491047A (en) | Multi-label text classification method and device, electronic equipment and storage medium | |
| CN111985194A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN112328656B (en) | Service query method, device, equipment and storage medium based on middle platform architecture | |
| CN113434542A (en) | Data relation identification method and device, electronic equipment and storage medium | |
| CN113313211A (en) | Text classification method and device, electronic equipment and storage medium | |
| CN115314570B (en) | Data issuing method, device, equipment and medium based on protocol development framework | |
| CN114817408B (en) | Scheduling resource identification method and device, electronic equipment and storage medium | |
| CN113918305B (en) | Node scheduling method, node scheduling device, electronic equipment and readable storage medium | |
| CN111553133B (en) | Report generation method and device, electronic equipment and storage medium | |
| CN114490137A (en) | Service data real-time statistical method and device, electronic equipment and readable storage medium | |
| CN113918296A (en) | Model training task scheduling execution method and device, electronic equipment and storage medium | |
| CN113449037A (en) | AI-based SQL engine calling method, device, equipment and medium | |
| CN111651167A (en) | Method and device for identifying dependency relationship of scheduling task and computer readable storage medium | |
| CN114819590B (en) | Policy intelligent recommendation method, device, equipment and storage medium | |
| CN115174698B (en) | Market data decoding method, device, equipment and medium based on table entry index |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |