CA3168298A1 - Method for data query based on ignite cache architecture and system thereof - Google Patents
Method for data query based on ignite cache architecture and system thereofInfo
- Publication number
- CA3168298A1 CA3168298A1 CA3168298A CA3168298A CA3168298A1 CA 3168298 A1 CA3168298 A1 CA 3168298A1 CA 3168298 A CA3168298 A CA 3168298A CA 3168298 A CA3168298 A CA 3168298A CA 3168298 A1 CA3168298 A1 CA 3168298A1
- Authority
- CA
- Canada
- Prior art keywords
- query
- cache
- data
- ignite
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000003213 activating effect Effects 0.000 claims description 18
- 230000002688 persistence Effects 0.000 claims description 5
- 238000013459 approach Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A method and system for data query based on an Ignite cache architecture, the Ignite cache architecture being provided with an intermediate cache layer effectively increases the querying rate. The method comprises: step S1, providing a caching algorithm, temporarily storing query objects produced on the basis of the caching algorithm into a buffer pool according to the order of priority; step S2, starting a calling task on the basis of the query objects to extract cached data from an external data source and saving in Ignite distributed nodes; and, step S4, receiving a query request launched by a user, when a query object matching the query request is stored in the Ignite distributed nodes, then associating corresponding cached data and outputting as a feedback; otherwise, directly accessing the external data source to acquire result data matching the query object and outputting as a feedback. The system comprises the method mentioned in the described solution.
Description
METHOD FOR DATA QUERY BASED ON IGNITE CACHE ARCHITECTURE AND
SYSTEM THEREOF
BACKGROUND OF THE INVENTION
Technical Field [0001] The present invention relates to the technical field of data processing, and more particularly to a method for data query based on Ignite cache architecture and a system thereof.
Description of Related Art
SYSTEM THEREOF
BACKGROUND OF THE INVENTION
Technical Field [0001] The present invention relates to the technical field of data processing, and more particularly to a method for data query based on Ignite cache architecture and a system thereof.
Description of Related Art
[0002] With the development of the business scales of internet-related companies, querying massive data usually takes considerable time. In particular, for complicated hotspot data query, high concurrency often means great pressure on the computing layer and the storage layer, and tends to lead to delayed query results. Additionally, in practical applications, storage of underlying data in complicated projects can involve various external data source, such as RDBMS, NoSQL, and HDFS. Since the storage media are diverse, different query instructions are touted to different external data sources. In the prior art, for accelerating query results, one common solution is to cache hotspot data in each data storage layer itself. However, such a known solution is only suitable for queries of real-time hotspot data. If a user wants to query non-hotspot data, such as same-period data, he/she has to additionally access external data sources. In this case, by far, the query efficiency is still low.
SUMMARY OF THE INVENTION
SUMMARY OF THE INVENTION
[0003] The objective of the present invention is to provide a method for data query based on Date Regue/Date Received 2022-07-18 Ignite cache architecture and a system thereof, which use an intermediate cache layer to effectively speed up the query process.
[0004] In order to achieve the foregoing objective, in one aspect, the present invention provides a method for data query based on Ignite cache architecture, which comprises:
[0005] Si, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
[0006] S2, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node;
[0007] S4, receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting;
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
[0008] Preferably, between S2 and S4, the method further comprises:
[0009] S3: estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
[0010] More preferably, in Si, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order comprises:
[0011] using the cache algorithm to identify the query objects having high query frequencies and long query elapses; and
[0012] caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
[0013] Preferably, in S2, based on the query objects, activating a scheduling task so as to extract Date Regue/Date Received 2022-07-18 the cache data from corresponding external data sources and storing the data into an Ignite distributed node comprises:
[0014] creating plural storage tables for caching data sets using SQL syntax by means of creating classes; and
[0015] based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node comprises at least one storage table.
[0016] Further, after the step of, based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, the method further comprises:
[0017] performing a cache persistence operation on the cache data in the node, so that the cache data can be automatically loaded in a memory of the node during reactivating services.
[0018] Preferably, the step of, when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting comprises:
[0019] according to the query request initiated by the user, matching query objects from the corresponding node; and
[0020] associating and matching the query objects from the different external data sources, performing analysis using the SQL syntax, and then outputting feed-back results.
[0021] As compared to the prior art, the method for data query based on Ignite cache architecture of the present invention provides the following beneficial effects:
[0022] The method for data query based on Ignite cache architecture of the present invention begins with the step of setting a cache algorithm. Then query objects are acquired from the corresponding external data sources and cached in a buffer pool in a priority order.
Afterward, different storage tables are created using the SQL syntax by means of creating Date Regue/Date Received 2022-07-18 classes for caching data sets. This step is essentially about creating a cache intermediate layer. Further, according to the acquired query objects, scheduling tasks are activated to extract the cache data from the corresponding external data sources. The extracted data are then stored in storage tables of nodes so that in response to a query request from a user, the method determines whether there is any query object matching the query request existing in the cache intermediate layer. If yes, it indicates that the query hits a target, and the corresponding cache data are associated and output as feedback. If there is not any matching query object, it means that the query does not hit any target. In this case, a query can be made using the known OLAP approach to directly access the external data sources to acquire result data matching the query object and feedback the result data by outputting.
Afterward, different storage tables are created using the SQL syntax by means of creating Date Regue/Date Received 2022-07-18 classes for caching data sets. This step is essentially about creating a cache intermediate layer. Further, according to the acquired query objects, scheduling tasks are activated to extract the cache data from the corresponding external data sources. The extracted data are then stored in storage tables of nodes so that in response to a query request from a user, the method determines whether there is any query object matching the query request existing in the cache intermediate layer. If yes, it indicates that the query hits a target, and the corresponding cache data are associated and output as feedback. If there is not any matching query object, it means that the query does not hit any target. In this case, a query can be made using the known OLAP approach to directly access the external data sources to acquire result data matching the query object and feedback the result data by outputting.
[0023] It is thus clear that with the method of the present invention, cache data from various types of external data sources can be drawn into the cache intermediate layer in advance and stored in the Ignite distributed nodes. The cache intermediate layer can, downward, extract the cache data in different types of external data sources, including RDBMS, NoSQL, HDFS, etc., and can, upward, provide unified query services to users.
Thereby, when a user initiates a query request, if the query hits a target in the cache intermediate layer, the corresponding cache data can be directly associated and output as feedback, so as to significantly improve query efficiency; and if the query does not hit any target in the cache intermediate layer, the known OLAP approach can be used to generate an output for the query, thereby ensuring feedback of query results.
Thereby, when a user initiates a query request, if the query hits a target in the cache intermediate layer, the corresponding cache data can be directly associated and output as feedback, so as to significantly improve query efficiency; and if the query does not hit any target in the cache intermediate layer, the known OLAP approach can be used to generate an output for the query, thereby ensuring feedback of query results.
[0024] In another aspect, the present invention provides a system for data query based on Ignite cache architecture, used in the method for data query based on Ignite cache architecture of the foregoing technical scheme the. The system comprises:
[0025] an algorithm setting unit, for setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
[0026] a cache creating unit, for, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache Date Regue/Date Received 2022-07-18 data into an Ignite distributed node;
[0027] a result outputting unit, for receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting;
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
[0028] Preferably, further comprises:
[0029] a cache updating unit, for estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
[0030] More preferably, the algorithm setting unit comprises:
[0031] a query object module, for identifying the query objects having high query frequencies and long query elapses;
[0032] an identifying module, for caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
[0033] Preferably, the cache creating unit comprises:
[0034] a storage table creating module, for creating plural storage tables for caching data sets using SQL syntax by means of creating classes; and
[0035] a cache data extracting module, for based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
[0036] As compared to the prior art, the disclosed system for data query based on Ignite cache architecture provides beneficial effects that are similar to those provided by the disclosed method for data query based on Ignite cache architecture as enumerated above, and thus Date Regue/Date Received 2022-07-18 no repetitions are made herein.
BRIEF DESCRIPTION OF THE DRAWINGS
BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings are provided herein for better understanding of the present invention and form a part of this disclosure. The illustrative embodiments and their descriptions are for explaining the present invention and by no means form any improper limitation to the present invention, wherein:
[0038] FIG. 1 is a flowchart of a method for data query based on Ignite cache architecture according to Embodiment 1 of the present invention;
[0039] FIG. 2 is a block diagram of the method of Embodiment 1 of the present invention showing related components; and
[0040] FIG. 3 is a structural diagram of a system for data query based on Ignite cache architecture according to Embodiment 2 of the present invention.
[0041] List of reference numbers:
[0042] 1- algorithm setting unit, 2- cache creating unit;
[0043] 3- cache updating unit, 4- result outputting unit.
DETAILED DESCRIPTION OF THE INVENTION
DETAILED DESCRIPTION OF THE INVENTION
[0044] To make the foregoing objectives, features, and advantages of the present invention clearer and more understandable, the following description will be directed to some embodiments as depicted in the accompanying drawings to detail the technical schemes disclosed in these embodiments. It is, however, to be understood that the embodiments referred herein are only a part of all possible embodiments and thus not exhaustive. Based on the embodiments of the present invention, all the other embodiments can be conceived without creative labor by people of ordinary skill in the art, and all these and other embodiments shall be encompassed in the scope of the present invention.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
[0045] Embodiment 1
[0046] Referring to FIG. 1, the present embodiment provides a method for data query based on Ignite cache architecture, which comprises:
[0047] Si, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order; S2, based on query object activating scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node; and S4, receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting; otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
[0048] The method for data query based on Ignite cache architecture of the present embodiment, as shown in FIG. 2, begins with the step of setting a cache algorithm. Then query objects are acquired from the corresponding external data sources and cached in a buffer pool in a priority order. Afterward, different storage tables are created using the SQL syntax by means of creating classes for caching data sets. This step is essentially about creating a cache intermediate layer. Further, according to the acquired query objects, scheduling tasks are activated to extract the cache data from the corresponding external data sources.
The extracted data are then stored in storage tables of nodes so that in response to a query request from a user, the method determines whether there is any query object matching the query request existing in the cache intermediate layer. If yes, it indicates that the query hits a target, and the corresponding cache data are associated and output as feedback. If there is not any matching query object, it means that the query does not hit any target. In this case, a query can be made using the known OLAP approach to directly access the external data sources to acquire result data matching the query object and feedback the result data by outputting.
Date Regue/Date Received 2022-07-18
The extracted data are then stored in storage tables of nodes so that in response to a query request from a user, the method determines whether there is any query object matching the query request existing in the cache intermediate layer. If yes, it indicates that the query hits a target, and the corresponding cache data are associated and output as feedback. If there is not any matching query object, it means that the query does not hit any target. In this case, a query can be made using the known OLAP approach to directly access the external data sources to acquire result data matching the query object and feedback the result data by outputting.
Date Regue/Date Received 2022-07-18
[0049] It is thus clear that with the method of the present invention, cache data from various types of external data sources can be drawn into the cache intermediate layer in advance and stored in the Ignite distributed nodes. The cache intermediate layer can, downward, extract the cache data in different types of external data sources, including RDBMS, NoSQL, HDFS, etc., and can, upward, provide unified query services to users.
Thereby, when a user initiates a query request, if the query hits a target in the cache intermediate layer, the corresponding cache data can be directly associated and output as feedback, so as to significantly improve query efficiency; and if the query does not hit any target in the cache intermediate layer, the known OLAP approach can be used to generate an output for the query, thereby ensuring feedback of query results. In practical implementations, dynamic addition of external data sources can be simply achieved adding corresponding interfaces for the external data source services, so as to perfectly support transverse expansion of external data sources.
Thereby, when a user initiates a query request, if the query hits a target in the cache intermediate layer, the corresponding cache data can be directly associated and output as feedback, so as to significantly improve query efficiency; and if the query does not hit any target in the cache intermediate layer, the known OLAP approach can be used to generate an output for the query, thereby ensuring feedback of query results. In practical implementations, dynamic addition of external data sources can be simply achieved adding corresponding interfaces for the external data source services, so as to perfectly support transverse expansion of external data sources.
[0050] Optionally, in order to facilitate making queries, the present embodiment further involves providing a unified query services interface for the cache intermediate layers. With the query services interface, data in different external data sources can be queried in a unified manner, which means that users need not to care about the different query syntaxes of different storage media and data storage locations at all, and the desired query result can be easily obtained by calling the query services interface, thereby improving query efficiency and query experience.
[0051] For further expanding the range of data queries, in the embodiment described above, between S2 and S4, the method further comprises: S3: estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node. For example, if it is expected that there will be many users making the same query request about some hotspot data and the matching same-period data tomorrow, the cache data corresponding to the hotspot data and the matching same-period data can be acquired Date Regue/Date Received 2022-07-18 from the relevant external data sources and stored in the in Ignite distributed nodes in advance for later use. In this manner, once a user makes this expected query request, a query result can be generated very soon, and associated data from different sources can be provided as feedback at the same time. Moreover, this helps estimate the access volume caused by the query request tomorrow, so that the Ignite distributed nodes can update the related cache data in advance. Exemplarily, the cache data may be updated by means cleaning and insertion successively.
[0052] Specifically, in the embodiment described above, the step of Si, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order comprises:
[0053] using the cache algorithm to identify the query objects having high query frequencies and long query elapses; and caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is the sum of the weight values of query frequency and query elapse.
[0054] In practical implementations, the cache algorithm may have diverse cache strategies.
Preferably, the cache strategy is about targeting high-frequency and/or long-elapse objects as the query objects, and assigning weight values to them according to their respective frequencies and/or elapses. The weight results of the query objects are sorted in the TopN order and cached into a buffer pool. It is to be noted that the screening criteria like the query frequencies and/or elapses can be set according to applications in practical implementations, and the present embodiment makes no limitations thereto.
Preferably, the cache strategy is about targeting high-frequency and/or long-elapse objects as the query objects, and assigning weight values to them according to their respective frequencies and/or elapses. The weight results of the query objects are sorted in the TopN order and cached into a buffer pool. It is to be noted that the screening criteria like the query frequencies and/or elapses can be set according to applications in practical implementations, and the present embodiment makes no limitations thereto.
[0055] Further, in the embodiment described previously, the step of, based on the query objects, activating scheduling task so as to extract the cache data from corresponding external data sources and storing the data into an Ignite distributed node comprises:
[0056] creating plural storage tables for caching data sets using SQL syntax by means of creating classes; and based on the query objects, activating scheduling task, respectively, to extract Date Regue/Date Received 2022-07-18 the cache data from the corresponding external data source and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
[0057] Specifically, nodes are created through the process described blow.
First, setting is made through setIndexedTypes. Afterward, nodes are created using the SQL syntax supported by Ignite, such as CREATE TABLE CashA(FILED1 TNT, FIELD2 VARCHAR,FIELD3 VARCHAR, PRIMARY KEY (FIELD1)) WITH \ "BACKUPS=1, AFFINITY KEY=FILED1\ "). In addition, the table CashA may be set with an index and a primary key. CashA represents the query object of the cache data. Later queries made on the cache data for CashA can be operated using the SQL syntax. It is also to be noted that the number of the nodes may be adapted to the size of the cache data. For a small amount of cache data, only one node may be enough. In this case, all the cache data are stored in the storage table of the node. If the amount of cache data is relatively large, more nodes may be set. In this case, the cache data are distributed across the nodes. Storage of the cache data may be in the form of classes, such as Cache.put(1, new CashA
(1,2,3)).
Alternatively, an asynchronous form may be used, such as Cache. putAsync(1, new CashA (1,2,3)). The present embodiment makes no limitations thereto.
First, setting is made through setIndexedTypes. Afterward, nodes are created using the SQL syntax supported by Ignite, such as CREATE TABLE CashA(FILED1 TNT, FIELD2 VARCHAR,FIELD3 VARCHAR, PRIMARY KEY (FIELD1)) WITH \ "BACKUPS=1, AFFINITY KEY=FILED1\ "). In addition, the table CashA may be set with an index and a primary key. CashA represents the query object of the cache data. Later queries made on the cache data for CashA can be operated using the SQL syntax. It is also to be noted that the number of the nodes may be adapted to the size of the cache data. For a small amount of cache data, only one node may be enough. In this case, all the cache data are stored in the storage table of the node. If the amount of cache data is relatively large, more nodes may be set. In this case, the cache data are distributed across the nodes. Storage of the cache data may be in the form of classes, such as Cache.put(1, new CashA
(1,2,3)).
Alternatively, an asynchronous form may be used, such as Cache. putAsync(1, new CashA (1,2,3)). The present embodiment makes no limitations thereto.
[0058] Optionally, in the embodiment described previously, after the step of based on the query objects, activating scheduling tasks, respectively, to extract the cache data from the corresponding external data source and storing the cache data in one said node or in different said nodes, the method further comprises: performing a persistence operation on the cache data in the node, so that the cache data can be automatically loaded in a memory of the node during activation of services. In practical implementations, with the persistence operation for the Ignite cache data activated, the cache data are automatically loaded to the memory of the node during reboot of the services. If the persistence operation for the Ignite cache data is deactivated, reboot of the services causes loss of the cache data. It is understandable that the cache data in the node contain the entire data set Date Regue/Date Received 2022-07-18 including indicators and dimensions. The entire cluster comprises the whole data set.
[0059] Preferably, in the embodiment described previously, if the Ignite distributed node stores a query object matching the query request, the step of associating and feeding-back the corresponding cache data through outputting comprises:
[0060] according to the query request initiated by the user, identifying the matching query objects from the corresponding node; and associating and matching the query objects from the different external data sources before performing analysis using the SQL syntax, and then outputting feed-back results.
[0061] In practical business scenarios, a user usually needs to acquire cache data from different external data sources and then associate the queries. For addressing this problem, the present embodiment uses a cache intermediate layer for associated queries of data from different external data sources. For example, purchase order data are stored in ElasticSearch, and member data are stored in PostgreSql. When a user wants to acquire sales data of new and existing buyers, the cache intermediate layer caches the sales data from ElasticSearch to the node, and names this data set as Table A, and caches the member data from PostgreSql to the node, and name this data set as Table B. In response to a query request initiated by the user through the query services interface, the cache intermediate layer performs secondary processing on the cache data based on the query request instruction. In other words, the two parts of cache data from different data sources are associated and eventually only an SQL associated analysis result of Table A and Table Bis output. Hence, with the cache intermediate layer of the present embodiment, data logical processing across data sources can be conveniently achieved and the output query result is more accurate. Exemplarily, the secondary processing for the cache data comprises filtering of the cache results, screening of data rights, and secondary filtering after aggregation.
[0062] Embodiment 2 Date Regue/Date Received 2022-07-18
[0063] Referring to FIG. 1 and FIG. 3, the present embodiment provides a system for data query based on Ignite cache architecture, which comprises:
[0064] an algorithm setting unit 1, for setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
[0065] a cache creating unit 2, based on query object activating scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node; and
[0066] a result outputting unit 4, for receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cached data and feeding-back the same through outputting;
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
[0067] Preferably, the system further comprises:
[0068] a cache updating unit 3, for estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
[0069] Preferably, algorithm setting unit lcomprises:
[0070] a query object module, for identifying the query objects having high query frequencies and long query elapses;
[0071] an identifying module, for caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query duration.
[0072] Preferably, the cache creating unit 2 comprises:
[0073] a storage table creating module, for creating plural storage tables for caching data sets using SQL syntax by means of creating classes; and
[0074] a cache data extracting module, for based on the query objects, activating scheduling tasks to extract the cached data from the corresponding external data sources, respectively, Date Regue/Date Received 2022-07-18 and storing the cached data in one said node or in different said nodes, in which the node at least comprises one said storage table.
[0075] As compared to the prior art, the disclosed system for data query based on Ignite cache architecture provides beneficial effects that are similar to those provided by the disclosed method for data query based on Ignite cache architecture as enumerated in Embodiment 1, and thus no repetitions are made herein.
[0076] As will be appreciated by people of ordinary skill in the art, implementation of all or a part of the steps of the method of the present invention as described previously may be realized by having a program instruct related hardware components. The program may be stored in a computer-readable storage medium, and the program is about performing the individual steps of the methods described in the foregoing embodiments.
The storage medium may be a ROM/RAM, a hard drive, an optical disk, a memory card or the like.
The storage medium may be a ROM/RAM, a hard drive, an optical disk, a memory card or the like.
[0077] The present invention has been described with reference to the preferred embodiments and it is understood that the embodiments are not intended to limit the scope of the present invention. Moreover, as the contents disclosed herein should be readily understood and can be implemented by a person skilled in the art, all equivalent changes or modifications which do not depart from the concept of the present invention should be encompassed by the appended claims. Hence, the scope of the present invention shall only be defined by the appended claims.
Date Regue/Date Received 2022-07-18
Date Regue/Date Received 2022-07-18
Claims (10)
1. A method for data query based on Ignite cache architecture, comprising:
S1, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
S2, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node;
S4, receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting; otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
S1, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
S2, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node;
S4, receiving a query request initiated by a user, and when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting; otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
2. The method of claim 1, wherein between the steps S2 and S4 the method further comprises:
S3: estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
S3: estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
3. The method of claim 1 or 2, wherein in S 1, setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order comprises:
using the cache algorithm to identify the query objects having high query frequencies and long query elapses; and caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
using the cache algorithm to identify the query objects having high query frequencies and long query elapses; and caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
4. The method of claim 1, wherein in S2, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the data into Date Regue/Date Received 2022-07-18 an Ignite distributed node comprises:
creating plural storage tables for caching data sets using SQL syntax by means of creating classes;
and based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
creating plural storage tables for caching data sets using SQL syntax by means of creating classes;
and based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
5. The method of claim 4, wherein after the step of based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, the method further comprises:
performing a cache persistence operation on the cache data in the node, so that the cache data can be automatically loaded in a memory of the node during reactivating services.
performing a cache persistence operation on the cache data in the node, so that the cache data can be automatically loaded in a memory of the node during reactivating services.
6. The method of claim 1, wherein the step of when there is a said query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting comprises:
according to the query request initiated by the user, matching query objects from the corresponding node; and associating and matching the query objects from the different external data sources, performing analysis using the SQL syntax, and then outputting feed-back results.
according to the query request initiated by the user, matching query objects from the corresponding node; and associating and matching the query objects from the different external data sources, performing analysis using the SQL syntax, and then outputting feed-back results.
7. A system for data query based on Ignite cache architecture, comprising:
an algorithm setting unit, for setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
a cache creating unit, for, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node;
a result outputting unit, for receiving a query request initiated by a user, and when there is a said Date Regue/Date Received 2022-07-18 query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting;
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
an algorithm setting unit, for setting a cache algorithm, and caching query objects obtained using the cache algorithm into a buffer pool in a priority order;
a cache creating unit, for, based on the query objects, activating a scheduling task so as to extract the cache data from corresponding external data sources and storing the cache data into an Ignite distributed node;
a result outputting unit, for receiving a query request initiated by a user, and when there is a said Date Regue/Date Received 2022-07-18 query object matching the query request stored in the Ignite distributed node, associating the corresponding cache data and feeding-back the same through outputting;
otherwise, directly accessing the external data sources to acquire result data matching the query object and feeding-back the result data by outputting.
8. The system of claim 7, further comprising:
a cache updating unit, for estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
a cache updating unit, for estimating query requests from users, and regularly updating the cache data stored in the Ignite distributed node.
9. The system of claim 7, wherein the algorithm setting unit comprises:
a query object module, for identifying the query objects having high query frequencies and long query elapses; and an identifying module, for caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
a query object module, for identifying the query objects having high query frequencies and long query elapses; and an identifying module, for caching the query objects into the buffer pool based on weight results of the identified query objects according to a TopN sort order, in which the weight result is a sum of a weight of the query frequency and a weight of the query elapse.
10. The system of claim 7, wherein the cache creating unit comprises:
a storage table creating module, for creating plural storage tables for caching data sets using SQL
syntax by means of creating classes; and a cache data extracting module, for based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
Date Regue/Date Received 2022-07-18
a storage table creating module, for creating plural storage tables for caching data sets using SQL
syntax by means of creating classes; and a cache data extracting module, for based on the query objects, activating scheduling tasks to extract the cache data from the corresponding external data sources, respectively, and storing the cache data in one said node or in different said nodes, in which the node at least comprises one said storage table.
Date Regue/Date Received 2022-07-18
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910041334.6 | 2019-01-16 | ||
| CN201910041334.6A CN111444222A (en) | 2019-01-16 | 2019-01-16 | Data query method and system based on Ignite cache architecture |
| PCT/CN2019/106862 WO2020147334A1 (en) | 2019-01-16 | 2019-09-20 | Method and system for data query based on ignite cache architecture |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA3168298A1 true CA3168298A1 (en) | 2020-07-23 |
Family
ID=71614000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA3168298A Pending CA3168298A1 (en) | 2019-01-16 | 2019-09-20 | Method for data query based on ignite cache architecture and system thereof |
Country Status (3)
| Country | Link |
|---|---|
| CN (1) | CN111444222A (en) |
| CA (1) | CA3168298A1 (en) |
| WO (1) | WO2020147334A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113779452B (en) * | 2020-10-30 | 2024-04-16 | 北京沃东天骏信息技术有限公司 | Data processing method, device, equipment and storage medium |
| CN112751912B (en) * | 2020-12-15 | 2021-12-03 | 北京金山云网络技术有限公司 | Configuration adjustment method and device and electronic equipment |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101360260A (en) * | 2007-07-31 | 2009-02-04 | 广东新支点技术服务有限公司 | Hybrid mobile P2P content sharing system construction suitable for 2.5G/3G mobile data network |
| US20160259821A1 (en) * | 2015-03-04 | 2016-09-08 | General Electric Company | Efficient storage and searching of object state and relationships at a given point of time |
| CN105095520B (en) * | 2015-09-23 | 2018-07-27 | 电子科技大学 | The distributed memory database indexing means of structure-oriented data |
| CN105740472A (en) * | 2016-03-14 | 2016-07-06 | 中国科学院计算技术研究所 | Distributed real-time full-text search method and system |
| CN106326387B (en) * | 2016-08-17 | 2019-06-04 | 电子科技大学 | A kind of Distributed Storage structure and date storage method and data query method |
| CN106779974B (en) * | 2017-01-06 | 2020-12-29 | 重庆软岛科技股份有限公司 | Article transaction method and system |
| CN107040422B (en) * | 2017-04-25 | 2020-05-05 | 浙江工业大学 | Network big data visualization method based on materialized cache |
| CN108833500B (en) * | 2018-05-29 | 2021-03-30 | 创新先进技术有限公司 | Service calling method, service providing method, data transmission method and server |
-
2019
- 2019-01-16 CN CN201910041334.6A patent/CN111444222A/en active Pending
- 2019-09-20 WO PCT/CN2019/106862 patent/WO2020147334A1/en active Application Filing
- 2019-09-20 CA CA3168298A patent/CA3168298A1/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| CN111444222A (en) | 2020-07-24 |
| WO2020147334A1 (en) | 2020-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10387411B2 (en) | Determining a density of a key value referenced in a database query over a range of rows | |
| CN103729471B (en) | Data base query method and device | |
| US9183267B2 (en) | Linked databases | |
| US7512597B2 (en) | Relational database architecture with dynamic load capability | |
| US8229902B2 (en) | Managing storage of individually accessible data units | |
| US20100281005A1 (en) | Asynchronous Database Index Maintenance | |
| US20120215772A1 (en) | Grouping identity records to generate candidate lists to use in an entity and relationship resolution process | |
| US20080222114A1 (en) | Efficient directed acyclic graph representation | |
| US11449550B2 (en) | Ad-hoc graph definition | |
| US9171036B2 (en) | Batching heterogeneous database commands | |
| US20060288045A1 (en) | Method for aggregate operations on streaming data | |
| US20200320074A1 (en) | Filter Evaluation For Table Fragments | |
| CA3168298A1 (en) | Method for data query based on ignite cache architecture and system thereof | |
| CN105468644B (en) | Method and equipment for querying in database | |
| US20130003965A1 (en) | Surrogate key generation | |
| CN109614411B (en) | Data storage method, device and storage medium | |
| US20100191730A1 (en) | Efficiency in processing queries directed to static data sets | |
| US8200673B2 (en) | System and method for on-demand indexing | |
| CN109063215B (en) | Data retrieval method and device | |
| CN110781205A (en) | JDBC-based database direct-checking method, device and system | |
| US20160154812A1 (en) | Hybrid database management system | |
| CN102004800A (en) | Data query method and device of PDM (Product Data Management) system | |
| US20190179941A1 (en) | Materialized view generation | |
| KR20190129474A (en) | Apparatus and method for retrieving data | |
| US7483870B1 (en) | Fractional data synchronization and consolidation in an enterprise information system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |
|
| EEER | Examination request |
Effective date: 20220718 |