CN113282619A - Data rapid query method and system - Google Patents
Data rapid query method and system Download PDFInfo
- Publication number
- CN113282619A CN113282619A CN202110394441.4A CN202110394441A CN113282619A CN 113282619 A CN113282619 A CN 113282619A CN 202110394441 A CN202110394441 A CN 202110394441A CN 113282619 A CN113282619 A CN 113282619A
- Authority
- CN
- China
- Prior art keywords
- data
- cache
- request
- local cache
- frequency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 26
- 230000003993 interaction Effects 0.000 abstract description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a method and a system for quickly querying data, which comprise the following steps: calling data from a Redis cache according to a set frequency based on task scheduling, putting the data into a local cache, and generating a cache key name; reading data from a local cache according to a received data query request; if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters. According to the invention, the data interaction amount of the data request and the Redis cache is reduced through the local cache, and the use of network bandwidth can be reduced and the performance of application service is improved by adopting a mode of combining the local cache and the Redis cache.
Description
Technical Field
The invention relates to the technical field of data query, in particular to a method and a system for quickly querying data.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
The data size of a supply chain operation platform is ten million levels, data requests depend on a data center platform, most of the data requests are acquired in a data service API mode, the query efficiency of a database layer is uncontrollable, and the waiting time for directly acquiring data is long; in addition, data of a remote data center station changes according to frequency such as time, day or week according to configuration, and a traditional distributed cache (such as Spring + Redis) mode cannot effectively acquire the latest data; in addition, reids is used as a cache, the reids is used as a single cache server, and the distributed application system accesses the redis through the local area network; redis is assisted by a third party, network consumption exists, and if all data requests are accumulated in a Redis cache, the Redis cache can be greatly and concurrently accessed, and in the worst case, the Redis cache can be down.
Disclosure of Invention
In order to solve the problems, the invention provides a method and a system for quickly querying data, which are used for caching data in Redis according to request parameters by combining a local cache and the Redis cache in a ten-million-level data service API request mode, so that second-level display is realized.
In some embodiments, the following technical scheme is adopted:
a data quick query method comprises the following steps:
acquiring data from a data center station based on task scheduling and storing the data into a Redis cache; calling data from a Redis cache according to a set frequency, putting the data into a local cache, and generating a cache key name;
reading data from a local cache according to a received data query request;
if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
In other embodiments, the following technical solutions are adopted:
a system for fast querying data, comprising:
the data cache module is used for acquiring data from the data staging platform based on task scheduling and storing the data into a Redis cache; meanwhile, data are called from the Redis cache according to a set frequency, the data are placed into a local cache, and a cache key name is generated;
the data query module is used for reading data from a local cache according to a received data query request; if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
In other embodiments, the following technical solutions are adopted:
a terminal device comprising a processor and a memory, the processor being arranged to implement instructions; the memory is used for storing a plurality of instructions which are suitable for being loaded by the processor and executing the data fast query method.
In other embodiments, the following technical solutions are adopted:
a computer-readable storage medium, wherein a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor of a terminal device and executing the data fast query method.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, the data interaction amount of the data request and the Redis cache is reduced through the local cache, and the use of network bandwidth can be reduced and the performance of application service is improved by adopting a mode of combining the local cache and the Redis cache.
The local cache does not need a serialization process, and the speed is obviously improved. The availability of the system is improved, and the local cache can support for a period of time if the redis is down.
After the scheme of the invention is adopted, for the data volume of tens of millions of data scales, the data request is acquired by relying on a data center platform and adopting a data service API mode, and the data can be rapidly displayed.
Additional features and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
Fig. 1 is a flowchart of a data fast query method according to an embodiment of the present invention.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
In one or more embodiments, a method for fast querying data is disclosed, which specifically includes the following steps with reference to fig. 1:
(1) acquiring data from a data center station based on task scheduling and storing the data into a Redis cache; calling data from the Redis cache according to a set frequency, putting the data into a local cache, and generating a cache key value according to the request URL and the parameters;
in this embodiment, the execution frequency of task scheduling is set to be less than the highest frequency of data update of a data center, and a Redis cache is updated according to the frequency of task scheduling; such as: the highest frequency of data updating of a data center station is once every hour, and the execution frequency of task scheduling is set to be less than one hour, so that Redis cache can be updated in time; setting the validity period of the local cache, and automatically deleting the corresponding cached data (for example, deleting the cached data after one hour) after the validity period is reached; when the local cache does not exist, the local cache is obtained from Redis.
Before the task is scheduled and executed, the request time and frequency of a parameter list in a database table are checked, the request frequency of unit time is calculated, data which are not frequently used are deleted from the parameter list, and concurrency is reduced.
In this embodiment, the local cache does not need a serialization process, and the speed is obviously increased. The availability of the system is improved, and the local cache can support for a period of time if the redis is down. After the scheme is adopted, the data size is tens of millions of data volumes, the data request is acquired by depending on a data center platform and adopting a data service API mode, and the data is displayed quickly.
The data interaction amount of a system and a redis cache is reduced during data query through a local cache; distributed caching is realized through Redis, and local caching and the Redis caching are combined for use, so that the use of network bandwidth is reduced, and the performance of application service can be improved.
(2) Reading data from a local cache according to a received data query request; if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
In this embodiment, data to be cached is abstracted into a corresponding service api, and a set custom annotation is added to the data interface to determine whether caching is required. In the embodiment, data needing to be cached is abstracted into corresponding service api (or dao layer api), a specific custom annotation is added (various parameters can be added to the annotation to control various fine-grained accesses), when the interface is accessed, the value is taken in the local cache by using the Aspect of Spring AOP, if no value exists in the local cache, the data of Redis is read, and the data of Redis is refreshed to the local cache and is returned.
In order to further accelerate the data query speed, when the local cache receives a plurality of query conditions at the same time, the incidence relation of the plurality of query conditions is established, and the multi-condition data query result under any query condition combination is fed back according to the incidence relation.
As an alternative embodiment, a plurality of different types of parallel index libraries are arranged in the local cache, and the index libraries of corresponding types are used for retrieving data based on the types of the query conditions. Such as: establishing a related data index database based on different regions; when the region is limited in the query condition, the search is only carried out in the data index database corresponding to the corresponding region, and the whole cache database is not required to be traversed, so that the data query speed is further increased.
Example two
In one or more embodiments, a data fast query system is disclosed, which specifically includes:
the data cache module is used for acquiring data from the data staging platform based on task scheduling and storing the data into a Redis cache; meanwhile, data are called from the Redis cache according to a set frequency, the data are placed into a local cache, and a cache key name is generated;
the data query module is used for reading data from a local cache according to a received data query request; if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
It should be noted that specific implementation manners of the modules have been described in the first embodiment, and are not described herein again.
EXAMPLE III
In one or more embodiments, a terminal device is disclosed, which includes a server, where the server includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the data fast query method in the first embodiment. For brevity, no further description is provided herein.
It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may include both read-only memory and random access memory, and may provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.
In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software.
The data fast query method in the first embodiment may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, among other storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor. To avoid repetition, it is not described in detail here.
Those of ordinary skill in the art will appreciate that the various illustrative elements, i.e., algorithm steps, described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
Example four
In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and execute the data fast query method described in the first embodiment.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.
Claims (10)
1. A method for quickly querying data is characterized by comprising the following steps:
calling data from a Redis cache according to a set frequency based on task scheduling, putting the data into a local cache, and generating a cache key name;
reading data from a local cache according to a received data query request;
if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
2. The method for fast query of data according to claim 1, wherein before the task scheduling is executed, the request time and frequency of the parameter list in the database table are checked, the request frequency per unit time is calculated, and the infrequently used data are deleted from the parameter list to reduce the concurrency.
3. The method of claim 1, wherein the query is performed by a computer,
data needing to be cached is abstracted into corresponding service api, and set custom notes are added to the data interface to determine whether caching is needed.
4. The method as claimed in claim 1, wherein the Aspect of Spring AOP is utilized to take a value in the local cache, if no value exists in the local cache, the data in the Redis cache is read, and the data in the Redis cache is refreshed to the local cache while returning the data.
5. The method according to claim 1, wherein the execution frequency of the task scheduling is set to be less than the highest frequency of data update in the data, and the Redis cache is updated according to the frequency of the task scheduling; and setting the validity period of the local cache, and automatically deleting the corresponding cached data after the validity period is reached.
6. The method as claimed in claim 1, wherein when the local cache receives a plurality of query conditions at the same time, the local cache establishes an association relationship of the plurality of query conditions and feeds back a result of the query of the plurality of pieces of data in any combination.
7. The method as claimed in claim 6, wherein the local cache is provided with a plurality of parallel index libraries of different types, and the index libraries of corresponding types are used to retrieve data based on the type of the query condition.
8. A system for fast querying data, comprising:
the data cache module is used for acquiring data from the data staging platform based on task scheduling and storing the data into a Redis cache; meanwhile, data are called from the Redis cache according to a set frequency, the data are placed into a local cache, and a cache key name is generated;
the data query module is used for reading data from a local cache according to a received data query request; if the key value of the data to be inquired does not exist in the local cache, the data service API request is carried out in the Redis cache; simultaneously updating a request parameter queue in task scheduling, and recording request time and request frequency; and if the key value exists, returning corresponding data, and updating the use time and frequency of the corresponding parameters.
9. A terminal device comprising a processor and a memory, the processor being arranged to implement instructions; the memory is used for storing a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the data fast query method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a plurality of instructions, wherein the instructions are adapted to be loaded by a processor of a terminal device and to execute the method for fast data query according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110394441.4A CN113282619A (en) | 2021-04-13 | 2021-04-13 | Data rapid query method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110394441.4A CN113282619A (en) | 2021-04-13 | 2021-04-13 | Data rapid query method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113282619A true CN113282619A (en) | 2021-08-20 |
Family
ID=77276532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110394441.4A Pending CN113282619A (en) | 2021-04-13 | 2021-04-13 | Data rapid query method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113282619A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023241742A1 (en) * | 2022-06-14 | 2023-12-21 | 顺丰科技有限公司 | Redis-based performance adjustment method and apparatus, electronic device and medium |
| CN113806651B (en) * | 2021-09-18 | 2024-05-24 | 深圳市酷开网络科技股份有限公司 | Data caching method, device, server and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107025243A (en) * | 2016-02-02 | 2017-08-08 | 北京神州泰岳软件股份有限公司 | A kind of querying method of resource data, inquiring client terminal and inquiry system |
| CN110162543A (en) * | 2019-05-29 | 2019-08-23 | 北京奇艺世纪科技有限公司 | A kind of information query method and device |
| CN111563125A (en) * | 2020-05-08 | 2020-08-21 | 中国工商银行股份有限公司 | Data storage system, data query method and device |
| CN111581236A (en) * | 2020-04-02 | 2020-08-25 | 中国邮政储蓄银行股份有限公司 | Data query method and device |
-
2021
- 2021-04-13 CN CN202110394441.4A patent/CN113282619A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107025243A (en) * | 2016-02-02 | 2017-08-08 | 北京神州泰岳软件股份有限公司 | A kind of querying method of resource data, inquiring client terminal and inquiry system |
| CN110162543A (en) * | 2019-05-29 | 2019-08-23 | 北京奇艺世纪科技有限公司 | A kind of information query method and device |
| CN111581236A (en) * | 2020-04-02 | 2020-08-25 | 中国邮政储蓄银行股份有限公司 | Data query method and device |
| CN111563125A (en) * | 2020-05-08 | 2020-08-21 | 中国工商银行股份有限公司 | Data storage system, data query method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113806651B (en) * | 2021-09-18 | 2024-05-24 | 深圳市酷开网络科技股份有限公司 | Data caching method, device, server and storage medium |
| WO2023241742A1 (en) * | 2022-06-14 | 2023-12-21 | 顺丰科技有限公司 | Redis-based performance adjustment method and apparatus, electronic device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111651464B (en) | Data processing method, system and computing device | |
| US20150142928A1 (en) | Pre-fetching data | |
| CN111427859B (en) | Message processing method and device, electronic equipment and storage medium | |
| CN109977129A (en) | Multi-stage data caching method and equipment | |
| CN113032432A (en) | Data caching processing method, system, computing device and readable storage medium | |
| CN109241099B (en) | Data query method and terminal equipment | |
| CN113282619A (en) | Data rapid query method and system | |
| CN110659971B (en) | Transaction data processing method and device | |
| CN110737857A (en) | back-end paging acceleration method, system, terminal and storage medium | |
| CN111651374A (en) | Data processing method and device, computing equipment and readable storage medium | |
| CN107092628A (en) | The treating method and apparatus of time series data | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN111159160B (en) | Version rollback method and device, electronic equipment and storage medium | |
| CN108280134B (en) | Data circulation system and method based on account limit control, storage medium and terminal | |
| JP7341330B2 (en) | Date switching data offload method, device, equipment and media | |
| US11269784B1 (en) | System and methods for efficient caching in a distributed environment | |
| CN113076067B (en) | Method and device for eliminating cache data | |
| US20190065060A1 (en) | Caching instruction block header data in block architecture processor-based systems | |
| CN114528230A (en) | Cache data processing method and device and electronic equipment | |
| CN108234481B (en) | Method for controlling multi-machine distributed access to external system and distributed system | |
| CN110889053A (en) | Interface data caching method and device and computing equipment | |
| CN111209042A (en) | Method, device, medium and electronic equipment for establishing function stack | |
| CN117453707B (en) | Data updating method, device, electronic equipment and storage medium | |
| US12130867B2 (en) | Managing pre-caching of results | |
| CN118051284A (en) | Data calling method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210820 |