CN111858654A - Data query acceleration method and device based on memory calculation - Google Patents
Data query acceleration method and device based on memory calculation Download PDFInfo
- Publication number
- CN111858654A CN111858654A CN202010701955.5A CN202010701955A CN111858654A CN 111858654 A CN111858654 A CN 111858654A CN 202010701955 A CN202010701955 A CN 202010701955A CN 111858654 A CN111858654 A CN 111858654A
- Authority
- CN
- China
- Prior art keywords
- data
- memory database
- query
- hot
- extracted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data query acceleration method and equipment based on memory calculation, which are applied to a system comprising a data warehouse and a memory database, wherein the memory database stores hot data extracted from the data warehouse, and the method comprises the following steps: receiving a data query request sent by a user, determining a query result from the memory database according to an execution plan corresponding to the data query request, if result data corresponding to the data query request exists in the query result, returning the result data to the user and ending a query process corresponding to the data query request, so that the execution period of data query is shortened, the data query speed in a self-service analysis service system is increased, and the user interaction experience is improved.
Description
Technical Field
The present application relates to the field of big data processing, and more particularly, to a method and device for accelerating data query based on memory computation.
Background
The self-service analysis service system provides a self-service platform for data extraction and analysis for telecommunication operators. The platform utilizes the existing user data resources, and becomes available resources with commercial value through extraction and analysis. The self-service analysis service system supports service operation analysis in an operator and service requirements of users of the service operation analysis in self-service inquiry of consumption records, package service conditions and the like, is a self-service data statistics and analysis service platform, and business personnel perform operation processes of data model shelving, self-service data fetching, data association analysis, self-service statements and the like on the self-service analysis platform.
Fig. 1 is a schematic diagram illustrating a process of acquiring data of a self-service analysis service system in the prior art, where a certain province operator is taken as an example, users of system services cover the whole province and each city under the province. At present, the business and user data of each local city of the province operator are independently stored in a local data source built in each local city, so that the data are efficiently and quickly understood and processed, the data quality and the data integrity are ensured, a relational database of all business and user data of each city and county needs to be gathered by the operator, the requirements of personalized self-service query and analysis of users are met, and a system is ensured to be capable of finding out related data according to the query conditions of the users and feeding back the query results to the users.
The self-service analysis service access concurrency is high, complex query needs to be carried out from large data volume when self-service data fetching is caused by centralization of city data, part of small data is frequently loaded and displayed to the front end, the execution period is prolonged, the result feedback is delayed, the waiting time of a user is long, the service operation efficiency is influenced, the user interaction experience is poor, and the development of telecommunication operation business service is hindered.
How to improve the data query speed in the self-service analysis business system is a technical problem to be solved at present.
Disclosure of Invention
The invention provides a data query acceleration method and equipment based on memory calculation, which are used for solving the technical problems of long execution period of searching data and delayed result feedback in a self-service analysis service system in the prior art, and are applied to a system comprising a data warehouse and a memory database, wherein the memory database stores thermal data extracted from the data warehouse, and the method comprises the following steps:
receiving a data query request sent by a user;
determining a query result from the memory database according to an execution plan corresponding to the data query request;
and if the result data corresponding to the data query request exists in the query result, returning the result data to the user and ending the query process corresponding to the data query request.
Preferably, before receiving a data query request sent by a user, the method further includes:
receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
Preferably, the loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database specifically includes:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
and if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
Preferably, before loading the hot data to be extracted into the in-memory database according to the amount of hot data and the available storage space of the in-memory database, the method further comprises:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
Preferably, if the result data does not exist in the query result, the method further includes:
Querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
Correspondingly, the present invention also provides a data query acceleration device based on memory computing, which is applied to a system including a data warehouse and a memory database, wherein the memory database stores thermal data extracted from the data warehouse, and the device includes:
the receiving module is used for receiving a data query request sent by a user;
the determining module is used for determining a query result from the memory database according to the execution plan corresponding to the data query request;
and the return module is used for returning the result data to the user and ending the query process corresponding to the data query request if the result data corresponding to the data query request exists in the query result.
Preferably, the system further comprises a loading module, configured to:
receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
Determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
Preferably, the loading module is specifically configured to:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
and if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
Preferably, the loading module is further configured to:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
Preferably, the system further comprises a query module for:
querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a data query acceleration method and equipment for memory calculation, which are applied to a system comprising a data warehouse and a memory database, wherein the memory database stores hot data extracted from the data warehouse, receives a data query request sent by a user, determines a query result from the memory database according to an execution plan corresponding to the data query request, and returns the result data to the user and ends a query process corresponding to the data query request if the result data corresponding to the data query request exists in the query result, so that the execution period of data query is shortened, the data query speed in a self-service analysis service system is increased, and the user interaction experience is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 shows a prior art self-service analytics business system data acquisition process schematic;
fig. 2 is a schematic flow chart illustrating a method for accelerating data query based on memory computing according to a preferred embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for loading hot data into memory data according to an embodiment of the present invention;
fig. 4 is a flowchart illustrating a data query acceleration method based on memory computing according to another embodiment of the present invention;
FIG. 5 is a schematic diagram of a system according to the present invention in an embodiment of the present invention;
fig. 6 is a schematic structural diagram illustrating a data query acceleration apparatus based on memory computing according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As described in the background art, the self-service analysis service system is a self-service platform for providing data extraction and analysis for telecommunication operators, and a user can perform operation processes such as data model shelving, self-service data fetching, data association analysis, self-service report and the like on the platform.
The self-service analysis service access concurrency is high, complex query needs to be carried out from large data volume when the local data are centralized, part of small data are frequently loaded and displayed to the front end, the execution period is prolonged, the result feedback is delayed, and the service operation efficiency is influenced.
Therefore, the invention provides a data query acceleration method based on memory calculation, which is used for solving the problems of long execution period of searching data, delayed result feedback and low operation efficiency of self-service analysis service in the prior art.
Fig. 2 is a schematic flow chart of a data query acceleration method based on memory computing according to a preferred embodiment of the present invention, which is applied to a system including a data warehouse and a memory database, where the memory database stores hot data extracted from the data warehouse, receives a data query request sent by a user, determines a query result from the memory database according to an execution plan corresponding to the data query request, and returns result data to the user and ends a query process corresponding to the data query request if the query result includes result data corresponding to the data query request, where the method includes the following steps:
Step S201, receiving a data query request sent by a user.
The hot data extracted from the data warehouse is stored in the memory database, the data warehouse is a structured data environment of a decision support system and an online analysis application data source, and is used for researching and solving the problem of acquiring information from the database, the memory database is a database with data stored in the memory, and the hot data is online data which needs to be frequently accessed and inquired by a computing node.
In this step, the data query request is a query request sent by the user through the self-service analysis service platform, and the system receives the request.
In order to query the data more quickly, in a preferred embodiment of the present invention, before receiving a data query request sent by a user, the method further includes:
receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
The data stored and used by the system can be classified into two types, one type is divided into large data and small data according to the size of the data, the other type is divided into hot data and cold data according to the access frequency of the data, the hot data is online data which needs to be frequently accessed by the computing nodes, the access frequency requirement is high, the efficiency requirement is high, and the corresponding cold data is data which does not need to be frequently accessed.
The method loads the hot data with high access frequency into the memory database, and the memory database is established based on the memory, so that the high efficiency and timeliness of data read-write performance can be guaranteed.
In this step, a monitored thermal data event is received, the thermal data event is triggered according to the access frequency of each data page in the data warehouse, a thermal data access frequency threshold value can be preset, when the access frequency of the data page in the data warehouse reaches the thermal data access frequency threshold value, the thermal data event is triggered, then the thermal data volume of thermal data to be extracted in the corresponding data warehouse is determined according to the thermal data event, then the thermal data to be extracted is loaded into the memory database according to the thermal data volume and the available storage space of the memory database, and the rest non-thermal data are continuously stored in the data warehouse.
In addition, a temperature data access frequency threshold value can be set, temperature data is data with the access frequency of a data page being smaller than the access frequency of hot data in a preset time period, a data warehouse can be divided into two areas, one area stores the temperature data, the other area stores cold data, when the access frequency of the data page in the data warehouse reaches the temperature data access threshold value and does not reach the access frequency of the hot data, the corresponding data page and the data quantity are loaded to the temperature data storage area in the data warehouse, the cold data which does not reach the access frequency of the temperature data are loaded to the cold data storage area, the query priority of the data is achieved, and the data which are frequently accessed are queried and determined more quickly.
In order to more accurately load the hot data into the memory database, in a preferred embodiment of the present invention, the hot data to be extracted is loaded into the memory database according to the hot data amount and the available storage space of the memory database, specifically:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
and if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
Specifically, when the thermal data volume of the thermal data to be extracted is determined, whether the available storage space of the memory database is larger than the thermal data volume or not is judged, if yes, the thermal data to be extracted is directly loaded into the memory database, if not, the available storage space of the memory database is expanded and then the thermal data to be extracted is loaded into the memory database,
meanwhile, the system can monitor the access frequency of the hot data in the memory database in a preset time period in real time or periodically, and when the monitored access frequency of the hot data in the memory database is reduced to the preset hot data access frequency, the corresponding hot data is moved out of the memory database to a data warehouse and also to a set hot data storage area or a set cold data storage area, so that the use resources of the memory database are saved, the data distribution in the system is more reasonable, and the query is more efficient.
The memory nodes where the memory database is located can be one or more nodes in the system, idle storage nodes or standby storage nodes can be preset in the system, when the system judges that the amount of the thermal data to be accessed is larger than the available storage space of the memory database, expansion of the available storage space is executed, the idle storage nodes or the standby storage nodes are started, and if the idle storage nodes or the standby storage nodes are used up, networking is performed to store the thermal data to be extracted into the server.
It should be noted that the above scheme of expanding the available storage space and prioritizing hot data, warm data, and cold data according to access frequency of a data page is only one specific implementation manner in the present application, and other schemes of expanding the available storage space and prioritizing storage of data according to different access manners of data, such as online time of access and weights corresponding to different data types, all belong to the protection scope of the present application.
In order to more rapidly load the hot data to be fetched into the in-memory database, in a preferred embodiment of the present invention, before loading the hot data to be fetched into the in-memory database according to the amount of the hot data and the available storage space of the in-memory database, the method further includes:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
Specifically, in order to prevent the system from being blocked and occupying too much system resources due to too large hot data to be extracted, which is loaded to the memory database at one time, the hot data to be extracted is divided based on a preset data volume, the preset data volume is smaller than the hot data volume, a plurality of loading batches of the hot data to be extracted are determined according to a dividing result, and the hot data to be extracted are sequentially loaded to the memory database according to the determined loading batches according to the available storage space of the hot data volume memory database.
In addition, the hot data to be extracted can be divided into a plurality of loading batches, and the data can be divided into different loading batches according to different types, such as video, document, bill and the like.
The skilled in the art can flexibly set the loading batch with the preset data size or divide the loading batch according to different data types according to actual situations, which does not affect the protection scope of the present application.
Step S202, determining a query result from the memory database according to the execution plan corresponding to the data query request.
Specifically, after a data query request is received, a query process corresponding to the query request is triggered, the system analyzes and optimizes the SQL statements submitted in the query request to generate a corresponding execution plan, and then the system executes the execution plan to search from the in-memory database and determine a query result.
The execution plan may also be a plan which is generated in the data query request and can be directly executed, and when the system receives the data query request, the system directly opens a corresponding query process according to the execution plan to determine a query result from the in-memory database.
Step S203, if the result data corresponding to the data query request exists in the query result, returning the result data to the user and ending the query process corresponding to the data query request.
Specifically, the query process queries in the memory database, and if result data corresponding to the data query request exists, the result data is returned to the user, the corresponding query process is ended, and the system releases resources such as memory, calculation, network transmission and the like occupied by the query process.
In order to determine the query result more accurately, in a preferred embodiment of the present invention, if the result data does not exist in the query result, the method further includes:
querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
Specifically, if the result data does not exist in the query result of the query in the memory database, the query process queries whether the result data exists in the data warehouse according to the execution plan.
If the data warehouse is preset with a temperature data storage area and a cold data storage area, inquiring whether the temperature data storage area has result data or not, and inquiring the cold data storage area if the temperature data storage area does not have the result data, so as to judge whether the data warehouse has the result data or not.
And after the memory database and the data warehouse are queried, if no corresponding result data exists, returning an absent result, ending the query process and releasing resources occupied by the query process, such as memory, calculation, network transmission and the like.
The query process returns the result data to the user as long as the corresponding result data is queried in the whole system, and the operation is performed according to the next instruction of the user, and meanwhile, the system updates the access frequency of the accessed result data so as to determine whether the data is loaded into the memory database or the warm data storage area or the cold data storage area according to the access frequency of each data at regular intervals.
According to the technical scheme, the hot data extracted from the data warehouse is stored in the memory database, the data query request sent by a user is received, the query result is determined from the memory database according to the execution plan corresponding to the data query request, if the result data corresponding to the data query request exists in the query result, the result data are returned to the user, the query process corresponding to the data query request is ended, the execution period of data query is shortened, the data query speed in the self-service analysis service system is increased, and the user interaction experience is improved.
To further illustrate the technical idea of the present invention, fig. 3 shows a method for loading hot data into an in-memory database according to an embodiment of the present invention, which is applied to a system including a data warehouse and an in-memory database, and the method includes the following steps:
step S301, the system receives the thermal data event in real time.
Firstly, service data sources in a local-city relational database or a big data storage are collected to a local-city centralized relational data warehouse to form a uniform service big data pool.
Specifically, the system receives the thermal data event in real time, the thermal data event is triggered by the access frequency of each data page in the data warehouse, the system can monitor the access frequency of each data page in the data warehouse and the memory database within a preset time period in real time or periodically, the thermal data event is triggered when the access frequency reaches the preset thermal data access frequency, and the data page with the access frequency not reaching the preset thermal data access frequency is placed in the data warehouse.
In addition, a temperature data storage area and a cold data storage area can be arranged in the data warehouse, a preset temperature data access frequency is set, the preset temperature data access frequency is smaller than a preset hot data access frequency, data pages in the data warehouse are divided according to temperature data and cold data, so that query is carried out according to the sequence of the memory database, the temperature data storage area and the cold data storage area when query is carried out, and query feedback can be carried out on data query requests of users more quickly by dividing the data pages according to the access frequency.
And step S302, determining the thermal data amount of the thermal data to be extracted.
Specifically, after the system receives the thermal data event, the thermal data amount of the corresponding thermal data to be extracted in the data warehouse is determined according to the thermal data event.
Step S303, load the hot data to be extracted into the available storage space of the in-memory database.
Specifically, after the system determines the thermal data volume of the thermal data to be extracted, the system compares the storage space required by the thermal data volume with the available storage space of the memory database, loads the thermal data to be extracted into the memory database if the available storage space meets the storage space required by the thermal data volume, and loads the thermal data to be extracted into the memory database after expanding the available storage space of the memory database if the available storage space does not meet the storage space required by the thermal data volume.
The memory nodes where the memory database is located can be one or more nodes in the system, idle storage nodes or standby storage nodes can be preset in the system, when the system judges that the amount of the thermal data to be accessed is larger than the available storage space of the memory database, expansion of the available storage space is executed, the idle storage nodes or the standby storage nodes are started, and if the idle storage nodes or the standby storage nodes are used up, networking is performed to store the thermal data to be extracted into the server.
In addition, the system is connected with a self-service analysis system, specifically, the memory database is connected with the self-service analysis system, and fig. 5 is a schematic diagram of the system composition of the invention.
Through the technical scheme, the system receives the thermal data event in real time, extracts a large amount of data stored in the data warehouse, stores the thermal data to be extracted into the memory database for accelerated processing and analysis according to the thermal data quantity of the thermal data to be extracted and the available storage space of the memory database, and can quickly feed back the query result to a user, so that the response delay of the data query request is realized, and the service operation efficiency is improved.
In order to better increase the speed of data query, fig. 4 shows a schematic flow chart of a data query acceleration method based on memory computing according to another embodiment of the present invention, the method is applied to a system including a data warehouse and a memory database, the memory database stores thermal data extracted from the data warehouse, and the method includes the following steps:
step S401, receiving a data query request sent by a user.
Specifically, the user can send a corresponding access request, namely a data query request, through the front-end self-service analysis access service system.
Step S402, generating an execution plan based on the data query request.
Specifically, when the system receives a data query request, a corresponding query process is started, the data query request contains an SQL statement, and the system analyzes and optimizes the SQL statement in the query request and then generates a corresponding execution plan.
Step S403, the generated execution plan is executed in the in-memory database.
And step S404, judging whether result data meeting the query condition exist.
Specifically, when the system is executed in the memory database according to the execution plan, it is determined whether the memory database has result data meeting the query condition through the query process, if so, step S407 is executed, and if not, step S405 is executed.
Step S405, pushing down the execution plan to a data warehouse.
Specifically, when result data meeting the query conditions are not queried in the memory database, the execution plan is pushed down to the data warehouse for execution, and if a temperature data storage area and a cold data storage area are preset, query is performed in the order of the temperature data storage area and the cold data storage area.
And step S406, judging whether result data meeting the query condition exist in the data warehouse.
Specifically, it is determined whether result data meeting the query condition exists in the data warehouse, if yes, step S407 is executed, and if not, step S409 is executed.
And step S407, acquiring result data.
Specifically, the result data meeting the conditions is determined through the query process and then directly obtained.
And step S408, feeding back result data.
And step S409, finishing the inquiry.
Specifically, the query process is closed, and resources occupied by the query process, such as memory, calculation, network transmission, and the like, are released.
By applying the technical scheme, the data query request sent by the user is received, the data query request is analyzed to generate the execution plan, the execution plan is sequentially executed in the memory database and the data warehouse, the data warehouse is isolated from the hot data, the data query performance and efficiency are greatly improved by utilizing the processing characteristics of the memory database, and the problem of poor data query timeliness in a full-volume large data analysis scene is solved.
Corresponding to a data query acceleration method based on memory computation in a preferred embodiment of the present application, the present application further provides a data query acceleration device based on memory computation, which is applied to a system including a data warehouse and a memory database, where the memory database stores thermal data extracted from the data warehouse, as shown in fig. 6, the device includes:
A receiving module 601, configured to receive a data query request sent by a user;
a determining module 602, configured to determine a query result from the in-memory database according to an execution plan corresponding to the data query request;
a returning module 603, configured to, if result data corresponding to the data query request exists in the query result, return the result data to the user and end the query process corresponding to the data query request.
In a specific application scenario, the system further comprises a loading module, configured to:
receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
In a specific application scenario, the loading module is specifically configured to:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
And if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
In a specific application scenario, the loading module is further configured to:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
In a specific application scenario, the system further comprises a query module, configured to:
querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A method for accelerating data query based on memory computing, the method being applied to a system including a data warehouse and a memory database, the memory database storing therein hot data extracted from the data warehouse, the method comprising:
receiving a data query request sent by a user;
determining a query result from the memory database according to an execution plan corresponding to the data query request;
and if the result data corresponding to the data query request exists in the query result, returning the result data to the user and ending the query process corresponding to the data query request.
2. The method of claim 1, prior to receiving the data query request sent by the user, further comprising:
receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
3. The method according to claim 2, wherein the loading the hot data to be extracted into the memory database according to the amount of hot data and the available storage space of the memory database comprises:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
and if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
4. The method of claim 2, wherein prior to loading the hot data to be fetched into the in-memory database based on the amount of hot data and available storage space of the in-memory database, the method further comprises:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
5. The method of claim 1, wherein if the result data does not exist in the query result, the method further comprises:
querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
6. A data query acceleration apparatus based on memory computing, wherein the apparatus is applied to a system including a data warehouse and a memory database in which hot data extracted from the data warehouse is stored, the apparatus comprising:
the receiving module is used for receiving a data query request sent by a user;
the determining module is used for determining a query result from the memory database according to the execution plan corresponding to the data query request;
and the return module is used for returning the result data to the user and ending the query process corresponding to the data query request if the result data corresponding to the data query request exists in the query result.
7. The device of claim 6, further comprising a loading module to:
Receiving a monitored thermal data event, wherein the thermal data event is triggered according to the access frequency of each data page in the data warehouse;
determining a thermal data amount of thermal data to be extracted in the data warehouse according to the thermal data event;
and loading the hot data to be extracted into the memory database according to the hot data amount and the available storage space of the memory database.
8. The device of claim 7, wherein the loading module is further specifically configured to:
judging whether the available storage space is larger than the hot data volume;
if yes, loading the hot data to be extracted into the memory database based on the available storage space;
and if not, loading the hot data to be extracted into the memory database based on the expanded available storage space.
9. The device of claim 7, wherein the loading module is further to:
dividing the hot data to be extracted based on a preset data volume, wherein the preset data volume is smaller than the hot data volume;
and determining a plurality of loading batches of the hot data to be extracted according to the division result, and loading the hot data to be extracted into the memory database according to the hot data volume and the available storage space of the memory database according to the plurality of loading batches.
10. The device of claim 6, further comprising a query module to:
querying the data warehouse according to the execution plan to judge whether the result data exists in the data warehouse; and if the result data does not exist in the data warehouse, returning a notification of query failure to the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010701955.5A CN111858654A (en) | 2020-07-21 | 2020-07-21 | Data query acceleration method and device based on memory calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010701955.5A CN111858654A (en) | 2020-07-21 | 2020-07-21 | Data query acceleration method and device based on memory calculation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858654A true CN111858654A (en) | 2020-10-30 |
Family
ID=73002149
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010701955.5A Pending CN111858654A (en) | 2020-07-21 | 2020-07-21 | Data query acceleration method and device based on memory calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858654A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254477A (en) * | 2021-06-04 | 2021-08-13 | 威讯柏睿数据科技(北京)有限公司 | Data acquisition method and equipment based on memory |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968464A (en) * | 2012-11-08 | 2013-03-13 | 广东电子工业研究院有限公司 | Index-based local resource quick retrieval system and retrieval method thereof |
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN107133342A (en) * | 2017-05-16 | 2017-09-05 | 广州舜飞信息科技有限公司 | A kind of IndexR real-time data analysis storehouse |
| CN109344142A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
-
2020
- 2020-07-21 CN CN202010701955.5A patent/CN111858654A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968464A (en) * | 2012-11-08 | 2013-03-13 | 广东电子工业研究院有限公司 | Index-based local resource quick retrieval system and retrieval method thereof |
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN107133342A (en) * | 2017-05-16 | 2017-09-05 | 广州舜飞信息科技有限公司 | A kind of IndexR real-time data analysis storehouse |
| CN109344142A (en) * | 2018-08-22 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Data processing method, device, electronic equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254477A (en) * | 2021-06-04 | 2021-08-13 | 威讯柏睿数据科技(北京)有限公司 | Data acquisition method and equipment based on memory |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107784044B (en) | Table data query method and device | |
| CN111753016B (en) | Data processing method, device, system and computer readable storage medium | |
| CN111339171B (en) | Data query method, device and equipment | |
| CN110990372A (en) | Dimensional data processing method and device and data query method and device | |
| CN107679077B (en) | Paging implementation method and device, computer equipment and storage medium | |
| CN103207919A (en) | Method and device for quickly inquiring and calculating MangoDB cluster | |
| CN105975345B (en) | A kind of video requency frame data dynamic equalization memory management method based on distributed memory | |
| CN101247271A (en) | Performance data storage method and device | |
| CN110109948A (en) | Data query method, computer equipment and computer readable storage medium | |
| CN112035531A (en) | Sensitive data processing method, device, equipment and medium | |
| CN110347706A (en) | For handling method, Database Systems and the computer readable storage medium of inquiry | |
| CN111427931A (en) | Distributed query engine and method for querying relational database by using same | |
| CN115221186A (en) | Data query method, system and device and electronic equipment | |
| CN111858654A (en) | Data query acceleration method and device based on memory calculation | |
| CN115408384A (en) | Database access method, device and storage medium | |
| CN113065084B (en) | Data loading method and device, computer equipment and storage medium | |
| CN102724301B (en) | Cloud database system and method and equipment for reading and writing cloud data | |
| CN102325098B (en) | Group information acquisition method and system | |
| CN113037851B (en) | Method for cloud mobile phone system super-resolution based on storage implementation | |
| CN101963978A (en) | Distributed database management method, device and system | |
| CN113254480B (en) | Data query method and device | |
| CN107193749B (en) | Test method, device and equipment | |
| CN106549983B (en) | Database access method, terminal and server | |
| CN114745275A (en) | Node updating method and device in cloud service environment and computer equipment | |
| CN107451229B (en) | Database query method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201030 |
|
| RJ01 | Rejection of invention patent application after publication |