CN109684358B - Data query method and device - Google Patents
Data query method and device Download PDFInfo
- Publication number
- CN109684358B CN109684358B CN201710969976.3A CN201710969976A CN109684358B CN 109684358 B CN109684358 B CN 109684358B CN 201710969976 A CN201710969976 A CN 201710969976A CN 109684358 B CN109684358 B CN 109684358B
- Authority
- CN
- China
- Prior art keywords
- query
- data
- level cache
- request
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data query method and a data query device, which can dynamically analyze hot query, improve the cache hit rate and reduce the delay of query response by adopting a mode of coexistence of a first-level cache and a second-level cache, and improve the accuracy of cache data by utilizing a data updating mode of combining active cache and passive cache. The method comprises the following steps: inquiring data from the first-level cache according to the received inquiry request, and returning if the data is inquired; if the query is not overtime, the data is queried and returned from the second-level cache, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for querying data.
Background
With the continuous development of computer technology, more and more people choose to access a business system on line to acquire required information. For an online data query system with a large number of accesses, core business scenarios, such as online query of remaining tickets of train tickets, online query of remaining tickets of air tickets, online query of hotel room supply, and the like, generally have the following commonalities:
1. the query request is much larger than the write request; 2. interface service of an external query system needs to be called; 3. the processing power of the external query system is limited; 4. the change of the query entry parameters is large (for example, in a train ticket remaining query scene, a query site is not fixed, and the change is large).
For the core application module of the business system, the access amount is far larger than that of other links, and especially in the peak period of use, the performance and experience directly influence the subsequent benefit conversion. In the prior art, in order to improve the performance and user experience of the various core applications, a large number of queries are usually processed by establishing a data center, using a large cluster in combination with a crawler or a multi-channel provider, and the like. Specifically, the client triggers data query, the application server receives a query request, firstly queries the cache server, returns data if query data exists, and calls a provider query service to acquire data if the query data does not exist, and stores the data in the cache server and returns the data to the client after the data is successfully acquired. The whole process adopts a synchronization mechanism, and the caching strategy is passive caching.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
1) the cache hit rate is extremely low, and the delay of inquiring peak data response is large
Under the general condition, in an application scene of online data query, data query information needs to have relatively high timeliness, the caching time is not long, and daily, the hit rate of cached data is low under the design of the prior art scheme, a large number of query requests can directly request upstream query service, especially in a peak period of query and benefit conversion, a large number of HTTP links can be established, the burden of a service system is greatly increased, partial query requests cannot acquire data, and the usability of the service system is influenced;
2) by adopting a synchronous mechanism, the anti-risk capability is poor, and the attack of professional malicious users cannot be basically coped with
In the prior art, a synchronization mechanism is adopted for data query. Meanwhile, in a common data query scene, the query authority of a user cannot be limited, for example, the user is required to log in to query the remaining tickets of the train tickets, so that query limitation cannot be made according to the user identification, and the limitation effect according to the source IP is relatively limited. Therefore, if a malicious user uses an automation program to simulate a normal request and generates a large number of query requests in a short time, the application server and the provider query service cannot be processed normally, and the availability of the business system is affected.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for data query, which can dynamically analyze hot queries, improve a cache hit rate and reduce a delay of query response by using a mode in which a first-level cache and a second-level cache coexist, and improve an accuracy of cache data and an usability of an access peak period query system by using a cache data update mode in which an active cache and a passive cache are combined.
To achieve the above object, according to an aspect of an embodiment of the present invention, a method for data query is provided.
The data query method of the embodiment of the invention comprises the following steps: inquiring data from the first-level cache according to the received inquiry request, and returning if the data is inquired; if the query is not overtime, the data is queried and returned from the second-level cache, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time.
Optionally, the invoking an external query service interface for data query includes: and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
Optionally, the performing asynchronous data query by using the message queue includes: sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
Optionally, before invoking the external query service interface for data query, the method further includes: and judging whether the query request belongs to an invalid request or not, and if so, not querying.
Optionally, the method further comprises: according to a preset counting period, counting hot query requests by using historical query request data, calling an external query service interface to perform data query according to the hot query requests, and updating the queried data into a first-level cache and/or a second-level cache.
To achieve the above object, according to another aspect of the embodiments of the present invention, an apparatus for data query is provided.
The data query device of the embodiment of the invention comprises: the first-level cache query module is used for querying data from the first-level cache according to the received query request and returning the data if the data is queried; and if the query is not overtime, returning the queried data, updating the queried data into a first-level cache and a second-level cache, and if the query is overtime, querying the data from the second-level cache and returning the data, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time.
Optionally, the call query module is further configured to: and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
Optionally, the call query module is further configured to: sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
Optionally, the call query module is further configured to: before calling an external query service interface to perform data query, judging whether the query request belongs to an invalid request, and if the query request belongs to the invalid request, not inquiring.
Optionally, the apparatus further comprises: and the active cache module is used for counting the hot query requests by using the historical query request data according to a preset counting period, calling an external query service interface to perform data query according to the hot query requests, and updating the queried data into the primary cache and/or the secondary cache.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an electronic apparatus.
An electronic device of an embodiment of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for querying the data of the embodiment of the invention.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer-readable medium.
A computer-readable medium of an embodiment of the present invention has a computer program stored thereon, and when the program is executed by a processor, the program implements a method of data query of an embodiment of the present invention.
One embodiment of the above invention has the following advantages or benefits: because the technical means of combining the first-level cache and the second-level cache is adopted, after a server side for data query receives a query request, the server side firstly queries data from the first-level cache, and if the query is received, the server side returns the data; if not, calling an inquiry service interface provided by the provider of each external channel, and providing inquiry service by the provider of each external channel; if the query service provided by the external provider is overtime, the data is queried from the second-level cache in the query system and returned, that is, the second-level cache in the query system is used as the last way for querying the data, so that the technical problem of low cache hit rate in the prior art is solved, and the technical effects of improving the cache hit rate and reducing the delay of the data in the query peak period are achieved; by establishing an inquiry routing layer, dynamically switching the request mode to be synchronous or asynchronous according to the inquiry request amount in unit time of the current inquiry system, and carrying out data inquiry by combining measures such as a message queue and the like, the usability of the service system can be improved, the risk resistance of the service system is improved, and the attack of malicious users can be effectively responded; before calling an external query service interface to perform data query, judging whether a query request belongs to an invalid request, if so, not querying, thereby effectively reducing unnecessary performance consumption; by adopting the combination of updating the result obtained by calling the external query service according to the query request to the cache (namely, passive cache) and actively calling the external query service according to the hot query request and updating the cache (namely, active cache) according to the query result, the accuracy of the cache data can be improved, the usability of the system is improved, and the pressure of the external query service is relieved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main steps of a method of data query according to an embodiment of the invention;
FIG. 2 is an architectural diagram of a query system suitable for implementing the method of data querying of an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a method of data querying according to an embodiment of the invention;
FIG. 4 is a schematic diagram of caching logic for a method of data querying, according to an embodiment of the invention;
FIG. 5 is a schematic diagram of the synchronous and asynchronous processing mechanisms of query requests of a method of data querying according to an embodiment of the invention;
FIG. 6 is a schematic diagram of the main modules of an apparatus for data querying, according to an embodiment of the present invention;
FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As described above, for the online data query system with a large number of accesses, the core service scenarios thereof, such as online query of remaining tickets of train tickets, online query of remaining tickets of air tickets, online query of hotel room supply, and the like, are much larger than other links in access amount, and especially in a peak use period, the performance and experience thereof directly affect the subsequent benefit conversion.
Taking a business system for selling train tickets on line as an example, the core process can be abstracted into a plurality of scenes such as surplus ticket query, ticket buying, ticket refunding and the like, wherein the surplus ticket query is an important link and has some particularity: the access amount is far larger than that of other links, and particularly in the peak period of ticket buying, the performance and experience of the system directly influence the subsequent order placing process; the user does not need to log in when the remaining tickets are inquired, so that the possibility that the remaining tickets are attacked by a malicious user by using an automatic script and further the use of a normal user is influenced is higher.
In view of the above situation, the embodiment of the present invention provides an effective scheme that user experience, risk resistance, cost, and universality are more optimized than those of the prior art, so that cache hit rate can be improved, and query response delay can be reduced; and the usability of the service system can be improved, the risk resistance of the service system is improved, and the attack of malicious users can be effectively responded.
In the following description, for convenience of description, an application scenario of train ticket remainder inquiry will be taken as an example to describe in detail a processing scheme of a data inquiry request with a large data volume, but it does not mean that the technical scheme of data inquiry in the embodiment of the present invention is only applicable to this application scenario. The application scenarios in accordance with the situations that the query request is far greater than the write request, the interface service of the external query system needs to be called, the processing capability of the external query system is limited, and the query parameter has large change can all be perfected by using the technical scheme of the embodiment of the invention, and all belong to the protection scope of the embodiment of the invention.
Fig. 1 is a schematic diagram of main steps of a data query method according to an embodiment of the present invention, and as shown in fig. 1, the data query method according to the embodiment of the present invention mainly includes the following steps:
step S101: inquiring data from the first-level cache according to the received inquiry request, and returning if the data is inquired; otherwise, go to step S102. The present step is mainly to implement that after receiving the query request, the data is first obtained from the cache, and if the data cannot be obtained from the cache, the external query service interface is called from step S102 to perform data query, so that the access pressure of the subsequent external query service can be reduced.
Step S102: and continuing to call the external query service interface for data query, returning the queried data if the query is not overtime, updating the queried data into a first-level cache and a second-level cache, and querying and returning the data from the second-level cache if the query is overtime, wherein the first-level cache has a first preset data expiration deletion time, the second-level cache has a second preset data expiration deletion time, and the first preset data expiration deletion time (which can be set to 20 seconds for example and not limited) is less than the second preset data expiration deletion time (which can be set to 2 hours for example and not limited). The data expiration deletion time means that after the data is updated to the cache, the data is automatically deleted after the time stored in the cache exceeds the expiration deletion time.
As can be seen from the foregoing description, in order to balance the high hit rate of the cache and the high accuracy of the query data, the following query strategy is adopted in the embodiment of the present invention: in the embodiment of the invention, the first-level cache and the second-level cache belong to cache modules in the query system, and the external query service interface refers to a query service interface provided by each channel provider outside the query system. The interface is invoked when no data is queried in the first level cache. But if the query is overtime when the external query service interface is called for query service, the query service of the provider is abandoned, and data is acquired from a secondary cache inside the query system. In the embodiment of the present invention, the second-level cache is used as the last data obtaining path, on one hand, the data stored in the second-level cache is less accurate than the data stored in the first-level cache, and on the other hand, the data stored in the second-level cache is longer than the data stored in the first-level cache (i.e., the second preset data set in the second-level cache is longer in expiration deletion time, and the cache hit rate is higher than that of the first-level cache).
In the embodiment of the present invention, invoking the external query service interface to perform data query may be performed in a manner of combining synchronization and asynchronization, specifically: and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
The data asynchronous query by using the message queue comprises the following steps: sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
In this embodiment of the present invention, before invoking the external query service interface to perform data query, the method for querying data in this embodiment of the present invention may further include: and judging whether the query request belongs to an invalid request or not, and if so, not querying, thereby reducing unnecessary performance loss of the service system.
In addition, the data query method of the embodiment of the present invention may further include actively caching: according to a preset counting period, counting hot query requests by using historical query request data, calling an external query service interface to perform data query according to the hot query requests, and updating the queried data into a first-level cache and/or a second-level cache. Therefore, the accuracy of the data in the cache can be effectively guaranteed, and the user experience is improved.
FIG. 2 is an architectural diagram of a query system suitable for implementing the method of data querying of an embodiment of the present invention; fig. 3 is a schematic diagram of a main flow of a method of data query according to an embodiment of the present invention. The following describes the data query method according to the embodiment of the present invention in detail with reference to fig. 2 and 3, taking an application scenario of train ticket remainder query as an example.
As shown in fig. 2, a system suitable for implementing the data query method according to the embodiment of the present invention can be mainly divided into several layers, i.e., a client, a cache database, a server, and an external query service. The system supports horizontal smooth expansion while the availability is guaranteed, the number of services can be freely increased or decreased according to the actual online situation, the other parts also support cluster processing, and the system has high expandability. As shown in fig. 3, the method for querying data according to the embodiment of the present invention mainly includes the following processes: the first-level cache and the second-level cache are realized through a cache database, the cache data is updated through an active cache and a passive cache, and the query request is synchronously and/or asynchronously processed through a request routing layer and a message queue.
FIG. 4 is a diagram illustrating caching logic of a method for data querying, according to an embodiment of the present invention. As shown in fig. 4, in the embodiment of the present invention, the active cache plus the passive cache and the first-level cache plus the second-level cache are implemented by a cache database. The cache database In the embodiment of the present invention may be, but is not limited to, Redis (Redis is an open-source written using ANSI C language, supports network, may be based on Memory or persistent log-type, Key-Value database, and provides API for multiple languages), or a clustering scheme for Redis, for example, a native Redis-based clustering scheme In kyoto, a jimdb (jingdong In Memory database).
Firstly, the system defines a first-level cache and a second-level cache. The effective time of the aging cache (i.e. the first-level cache) is short, the accuracy is high (for example, but not limited to, the first preset data expiration deletion time is set to be 20 seconds), and the effective time of the long-term cache (i.e. the second-level cache) is long, and the accuracy is low (for example, but not limited to, the second preset data expiration deletion time is set to be 2 hours). When the server receives the query request, the server queries the first-level cache first, and if the query request is not received, the server calls an external query service provided by a provider to acquire the query request. During peak access periods, if the provider query service process times out, the data in the secondary cache is directly retrieved. By arranging the secondary cache, normal inquiry and ordering of the user in the peak period are ensured, and meanwhile, the condition that a large number of inquiry requests directly reach the inquiry service of a supplier to cause the inquiry service cannot be normally processed is avoided, so that the pressure of external inquiry service is reduced.
For the updating synchronization of the first-level cache and the second-level cache, on one hand, passive cache is adopted, the passive cache is triggered by a query request of a user, the passive cache is not limited to a hot query line, and the cache is updated after external query service of a supplier is called successfully; and on the other hand, the system is triggered by the system, a provider service interface is actively called according to a preset statistical period, and the information of the train number and the remaining tickets of the hot line is acquired and stored in a cache database.
FIG. 5 is a schematic diagram of the synchronous and asynchronous processing mechanisms of query requests of a method of data querying according to an embodiment of the present invention. As shown in fig. 5, in the embodiment of the present invention, the synchronous and asynchronous processing of the query request is implemented by the message queue.
After receiving the surplus ticket query request, the server first passes through a request routing layer, and the request routing layer selects a processing mode according to the current request amount (in the embodiment of the present invention, a cache database undertakes the work of recording the request amount, the request routing layer obtains the current request amount from the cache database, and simultaneously adds 1 to the request amount recorded in the cache database, the calculation mode of the current request amount may be, for example and without limitation, the query request amount in a unit time), if the request is an invalid request (in the embodiment of the present invention, the invalid request is an irregular request, and may mainly include 1, a parameter that some malicious users tamper at will, such as a parameter that a site (for example, beijing) tamps into some codes (for example, < script >), 2, some sites without a vehicle number (for example, east-beijing), and 3, a parameter that an irregular query line (for example, a line for which a system has not generated an order) is defined at peak time, and returns directly.
If the current request quantity can be synchronously processed by judging (for example, the query request quantity in unit time is not greater than a preset request quantity threshold value which can be flexibly set and adjusted according to the actual query capability and query requirements), the external query service of the supplier is directly called.
If the query request message is sent to the message queue (the message queue in the embodiment of the present invention may be, but is not limited to, activemq, but other components having a message queue function may also be selected, for example, JMQ (JMQ is a message middleware platform that is independently developed by a message group in kyoto and provides reliable message delivery and data, and has high availability, extensibility, and operation and maintenance)), and the client is notified of the query request to perform asynchronous processing, so that the client may acquire query result data by polling a query result interface.
When the message processor which is used for processing the query request in the asynchronous processing process monitors that the query request exists in the message queue, the external query service is called, and after the data is successfully acquired, the result data is cached to the application server.
Through the request routing layer and the message queue, the system can decouple the remaining ticket query process, and guarantee higher throughput of the system in the peak period, so that the processing capacity and the risk resistance of the system are improved.
The logic of the foregoing request routing layer selecting the processing mode of the query request according to the current request amount is summarized as follows: filtering invalid query requests, judging the current effective request quantity of the system, and carrying out asynchronization when the effective request quantity is larger than a preset request quantity threshold value or carrying out synchronization when the effective request quantity is smaller than the preset request quantity threshold value.
The following is an exemplary introduction to active caching: the hit query requests need to be counted first. In the embodiment of the invention, statistics is carried out according to historical query request data. For example, in the remaining ticket query application scenario, the hot line respectively counts all orders in the three time periods of the latest month and the latest week through the timed task program (i.e. the aforementioned preset statistical period, for example, but not limited to, one day is taken as a period), sorts the orders according to the number of orders from top to bottom, the three parts of data are weighted and reordered, the first 100 lines of station pairs (100 lines are not limited to the number, and the 100 lines of station pairs can be stored in a buffer memory as long as the number is consistent with (the number of station pairs is multiplied by the counted number of days multiplied by the average time length of single request processing)/the number of processing threads is less than the expiration deletion time of the second preset data in the secondary buffer memory, namely the 2 hours, and actively updating the query result to the first-level cache and the second-level cache.
Of course, the calculation algorithm of the hot line in the embodiment of the present invention may also adopt other calculation algorithms, for example, an empirical calculation mode set according to the calculation experience of the actual application scenario; the statistical data acquisition point can adopt ordering data or remaining ticket query data; the calculation method can adopt total amount, recent amount, or weighted average of several parts, etc.
The technical scheme of the data query of the embodiment of the invention can effectively improve the utilization rate of the query peak cache, greatly improve the user experience, further have an obvious effect on the improvement of the query and the later benefit conversion, effectively improve the processing capacity of the system on the conditions of peak/malicious attack and the like, and improve the availability and the usability of the system.
According to the technical scheme of the data query of the embodiment of the invention, as a technical means of combining the first-level cache and the second-level cache is adopted, the server side for performing data query firstly queries data from the first-level cache after receiving the query request, and returns the data if the data is queried; if not, calling an inquiry service interface provided by the provider of each external channel, and providing inquiry service by the provider of each external channel; if the query service provided by the external provider is overtime, the data is queried from the second-level cache in the query system and returned, that is, the second-level cache in the query system is used as the last way for querying the data, so that the technical problem of low cache hit rate in the prior art is solved, and the technical effects of improving the cache hit rate and reducing the delay of the data in the query peak period are achieved; by establishing an inquiry routing layer, the request mode is dynamically switched to be synchronous or asynchronous according to the inquiry request amount in unit time of the current system, and data inquiry is carried out by combining measures such as a message queue and the like, so that the availability of a service system can be improved, the risk resistance of the service system is improved, and the attack of malicious users can be effectively responded; before calling an external query service interface to perform data query, judging whether a query request belongs to an invalid request, if so, not querying, thereby effectively reducing unnecessary performance consumption; by adopting the combination of updating the result obtained by calling the external query service according to the query request to the cache (namely, passive cache) and actively calling the external query service according to the hot query request and updating the cache (namely, active cache) according to the query result, the accuracy of the cache data can be improved, the usability of the system is improved, and the pressure of the external query service is relieved.
Fig. 6 is a schematic diagram of main blocks of an apparatus for data query according to an embodiment of the present invention.
As shown in fig. 6, the apparatus 600 for data query according to the embodiment of the present invention mainly includes the following modules: a first-level cache query module 601 and a call query module 602.
The first-level cache query module 601 may be configured to query data from the first-level cache according to a received query request, and return the data if the data is queried; otherwise, triggering and calling the query module 602; the call query module 602 may be configured to continue to call an external query service interface for data query, return queried data if the query is not overtime, and update the queried data to a first-level cache and a second-level cache, and query and return data from the second-level cache if the query is overtime, where the first-level cache has a first preset data expiration deletion time, the second-level cache has a second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time.
Wherein the invoke query module 602 is further operable to: and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
Specifically, the asynchronous querying of data using the message queue may include: sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
In addition, the invoke query module 602 may be further operable to: before calling an external query service interface to perform data query, judging whether the query request belongs to an invalid request, and if the query request belongs to the invalid request, not inquiring.
In this embodiment of the present invention, the apparatus 600 may further include: and an active cache module (not shown in the figure) configured to count the hot query request according to a preset counting period by using the historical query request data, call an external query service interface to perform data query according to the hot query request, and update the queried data to the first-level cache and/or the second-level cache.
As can be seen from the above description, by adopting the technical means of combining the first-level cache and the second-level cache, the server side performing data query first queries data from the first-level cache after receiving the query request, and if the query is received, returns the data; if not, calling an inquiry service interface provided by the provider of each external channel, and providing inquiry service by the provider of each external channel; if the query service provided by the external provider is overtime, the data is queried from the second-level cache in the query system and returned, that is, the second-level cache in the query system is used as the last way for querying the data, so that the technical problem of low cache hit rate in the prior art is solved, and the technical effects of improving the cache hit rate and reducing the delay of the data in the query peak period are achieved; by establishing an inquiry routing layer, the request mode is dynamically switched to be synchronous or asynchronous according to the inquiry request amount in unit time of the current system, and data inquiry is carried out by combining measures such as a message queue and the like, so that the availability of a service system can be improved, the risk resistance of the service system is improved, and the attack of malicious users can be effectively responded; before calling an external query service interface to perform data query, judging whether a query request belongs to an invalid request, if so, not querying, thereby effectively reducing unnecessary performance consumption; by adopting the combination of updating the result obtained by calling the external query service according to the query request to the cache (namely, passive cache) and actively calling the external query service according to the hot query request and updating the cache (namely, active cache) according to the query result, the accuracy of the cache data can be improved, the usability of the system is improved, and the pressure of the external query service is relieved.
Fig. 7 shows an exemplary system architecture 700 of a data query method or data query apparatus to which embodiments of the present invention may be applied.
As shown in fig. 7, the system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 serves to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 701, 702, 703 to interact with a server 705 over a network 704, to receive or send messages or the like. The terminal devices 701, 702, 703 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).
The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 705 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 701, 702, 703. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the data query method provided by the embodiment of the present invention is generally executed by the server 705, and accordingly, the data query apparatus is generally disposed in the server 705.
It should be understood that the number of terminal devices, networks, and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a first level cache lookup module and a call lookup module. The names of these modules do not in some cases form a limitation on the modules themselves, for example, the first-level cache query module may also be described as a "module that queries data from the first-level cache according to a received query request".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: inquiring data from the first-level cache according to the received inquiry request, and returning if the data is inquired; if the query is not overtime, the data is queried and returned from the second-level cache, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time.
According to the technical scheme of the embodiment of the invention, as a technical means of combining the first-level cache and the second-level cache is adopted, after a server side for data query receives a query request, the server side firstly queries data from the first-level cache, and if the query is received, the server side returns the data; if not, calling an inquiry service interface provided by the provider of each external channel, and providing inquiry service by the provider of each external channel; if the query service provided by the external provider is overtime, the data is queried from the second-level cache in the query system and returned, that is, the second-level cache in the query system is used as the last way for querying the data, so that the technical problem of low cache hit rate in the prior art is solved, and the technical effects of improving the cache hit rate and reducing the delay of the data in the query peak period are achieved; by establishing an inquiry routing layer, the request mode is dynamically switched to be synchronous or asynchronous according to the inquiry request amount in unit time of the current system, and data inquiry is carried out by combining measures such as a message queue and the like, so that the availability of a service system can be improved, the risk resistance of the service system is improved, and the attack of malicious users can be effectively responded; before calling an external query service interface to perform data query, judging whether a query request belongs to an invalid request, if so, not querying, thereby effectively reducing unnecessary performance consumption; by adopting the combination of updating the result obtained by calling the external query service according to the query request to the cache (namely, passive cache) and actively calling the external query service according to the hot query request and updating the cache (namely, active cache) according to the query result, the accuracy of the cache data can be improved, the usability of the system is improved, and the pressure of the external query service is relieved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method of data querying, comprising:
inquiring data from the first-level cache according to the received inquiry request, and returning if the data is inquired; if not, then,
continuously acquiring the current total query request quantity from a cache database comprising a first-level cache and a second-level cache through a query routing layer to determine whether a query request is synchronously sent to a message queue to asynchronously call an external query service interface for data query, if the query is not overtime, returning the queried data, updating the queried data into the first-level cache and the second-level cache, and if the query is overtime, querying the data from the second-level cache and returning the data, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is smaller than the second preset data expiration deletion time;
according to a preset counting period, counting hot query requests by using historical query request data, calling an external query service interface to perform data query according to the hot query requests, and updating the queried data into a first-level cache and/or a second-level cache;
before calling an external query service interface to perform data query, the method further comprises: and judging whether the query request belongs to an invalid request or not, and if so, not querying.
2. The method of claim 1, wherein invoking an external query service interface for data queries comprises:
and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
3. The method of claim 2, wherein asynchronously querying data using the message queue comprises:
sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and
and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
4. An apparatus for querying data, comprising:
the first-level cache query module is used for querying data from the first-level cache according to the received query request and returning the data if the data is queried; otherwise, triggering the calling of the query module,
the system comprises a calling query module, a message queue, a query routing layer and a message queue, wherein the calling query module is used for continuously acquiring the current total query request quantity from a cache database comprising a first-level cache and a second-level cache through the query routing layer so as to determine whether to synchronously or send a query request to the message queue to asynchronously call an external query service interface for data query, if the query is not overtime, the queried data is returned, the queried data is updated to the first-level cache and the second-level cache, and if the query is overtime, the data is queried and returned from the second-level cache, wherein the first-level cache has first preset data expiration deletion time, the second-level cache has second preset data expiration deletion time, and the first preset data expiration deletion time is less than the second preset data expiration deletion time;
the active cache module is used for counting hot query requests by using historical query request data according to a preset counting period, calling an external query service interface to perform data query according to the hot query requests, and updating the queried data into a first-level cache and/or a second-level cache;
wherein the call query module is further configured to: before calling an external query service interface to perform data query, judging whether the query request belongs to an invalid request, and if the query request belongs to the invalid request, not inquiring.
5. The apparatus of claim 4, wherein the invoke query module is further configured to:
and judging whether the query request amount in unit time is greater than a preset request amount threshold value or not, if not, directly calling an external query service interface to perform data synchronous query, and if so, performing data asynchronous query by using a message queue.
6. The apparatus of claim 5, wherein the invoke query module is further configured to:
sending the query request to a message queue, and informing a client that the query request is asynchronous so that the client can poll to acquire a data asynchronous query result; and
and after monitoring the query request in the message queue, the message processor calls an external query service interface to perform data query according to the query request.
7. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-3.
8. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710969976.3A CN109684358B (en) | 2017-10-18 | 2017-10-18 | Data query method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710969976.3A CN109684358B (en) | 2017-10-18 | 2017-10-18 | Data query method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109684358A CN109684358A (en) | 2019-04-26 |
| CN109684358B true CN109684358B (en) | 2021-11-09 |
Family
ID=66183187
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710969976.3A Active CN109684358B (en) | 2017-10-18 | 2017-10-18 | Data query method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109684358B (en) |
Families Citing this family (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11681567B2 (en) | 2019-05-09 | 2023-06-20 | International Business Machines Corporation | Method and processor system for executing a TELT instruction to access a data item during execution of an atomic primitive |
| US11321146B2 (en) | 2019-05-09 | 2022-05-03 | International Business Machines Corporation | Executing an atomic primitive in a multi-core processor system |
| US20200356485A1 (en) * | 2019-05-09 | 2020-11-12 | International Business Machines Corporation | Executing multiple data requests of multiple-core processors |
| CN110287007A (en) * | 2019-05-20 | 2019-09-27 | 深圳壹账通智能科技有限公司 | Data call response method, server and computer readable storage medium |
| CN112055039B (en) * | 2019-06-06 | 2022-07-26 | 阿里巴巴集团控股有限公司 | Data access method, device and system and computing equipment |
| CN110290196B (en) * | 2019-06-20 | 2021-08-27 | 网宿科技股份有限公司 | Content distribution method and device |
| CN110413679B (en) * | 2019-07-31 | 2023-01-24 | 深圳前海微众银行股份有限公司 | Database information processing method, device, equipment and readable storage medium |
| CN110851477B (en) * | 2019-10-16 | 2022-09-09 | 浙江大搜车软件技术有限公司 | Stream data processing method, stream data processing device, computer equipment and storage medium |
| CN110837521A (en) * | 2019-11-15 | 2020-02-25 | 北京金山云网络技术有限公司 | Data query method and device and server |
| CN111026962B (en) * | 2019-12-03 | 2023-09-22 | 泰康保险集团股份有限公司 | Hierarchical relation query response time reduction method, server and user front end |
| CN110989939A (en) * | 2019-12-16 | 2020-04-10 | 中国银行股份有限公司 | Data cache processing method, device and equipment and cache component |
| CN111209467B (en) * | 2020-01-08 | 2023-05-26 | 中通服咨询设计研究院有限公司 | Data real-time query system in multi-concurrency multi-channel environment |
| CN111309251A (en) * | 2020-01-21 | 2020-06-19 | 青梧桐有限责任公司 | Data storage method, system, electronic device and readable storage medium |
| CN111400349A (en) * | 2020-02-19 | 2020-07-10 | 北京值得买科技股份有限公司 | Multilayer cache data acquisition method and device |
| CN111414383B (en) * | 2020-02-21 | 2024-03-15 | 车智互联（北京）科技有限公司 | Data request method, data processing system and computing device |
| CN111414198B (en) * | 2020-03-18 | 2023-05-02 | 北京字节跳动网络技术有限公司 | Request processing method and device |
| CN113535768A (en) * | 2020-04-15 | 2021-10-22 | 北京京东乾石科技有限公司 | Production monitoring method and device |
| CN111858086B (en) * | 2020-06-15 | 2023-07-21 | 福建天泉教育科技有限公司 | Processing method and storage medium for queue timeout in request task processing |
| CN111782698A (en) * | 2020-07-03 | 2020-10-16 | 广州探途网络技术有限公司 | Cache updating method and device and electronic equipment |
| CN111830039B (en) * | 2020-07-22 | 2021-07-27 | 南京认知物联网研究院有限公司 | Intelligent product quality detection method and device |
| CN111949857B (en) * | 2020-08-13 | 2024-06-11 | 中国民航信息网络股份有限公司 | Flight query request processing method and device and electronic equipment |
| CN112215453A (en) * | 2020-08-18 | 2021-01-12 | 汉海信息技术(上海)有限公司 | Inventory information processing method and device, electronic equipment and storage medium |
| CN113760633A (en) * | 2020-08-26 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method and device for monitoring interface service |
| CN112163001A (en) * | 2020-09-25 | 2021-01-01 | 同程网络科技股份有限公司 | High-concurrency query method, intelligent terminal and storage medium |
| CN112487047B (en) * | 2020-11-02 | 2024-08-13 | 广州明珞装备股份有限公司 | Spare part consumption data query method, system and storage medium |
| CN112463813A (en) * | 2020-11-19 | 2021-03-09 | 贝壳技术有限公司 | Data caching method and device, electronic equipment and storage medium |
| CN112749195A (en) * | 2020-12-30 | 2021-05-04 | 广州凡科互联网科技股份有限公司 | Asynchronous delay loading cache method |
| CN113778390B (en) * | 2021-01-14 | 2024-10-18 | 北京沃东天骏信息技术有限公司 | Interface access method and device |
| CN113114642B (en) * | 2021-03-30 | 2022-12-06 | 广州宸祺出行科技有限公司 | Interface integrated driver identity authentication method and device |
| CN113688129A (en) * | 2021-07-21 | 2021-11-23 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and device for encoding link transport data |
| CN113535782B (en) * | 2021-07-22 | 2024-05-07 | 成都数之联科技股份有限公司 | BI report query response method, system, electronic equipment and computer readable storage medium |
| CN113596177B (en) * | 2021-08-13 | 2023-06-27 | 四川虹美智能科技有限公司 | Method and device for resolving IP address of intelligent household equipment |
| CN114143376A (en) * | 2021-11-18 | 2022-03-04 | 青岛聚看云科技有限公司 | Server for loading cache, display equipment and resource playing method |
| CN114138840B (en) * | 2021-12-08 | 2024-09-10 | 中国建设银行股份有限公司 | Data query method, device, equipment and storage medium |
| CN116028525A (en) * | 2023-03-31 | 2023-04-28 | 成都四方伟业软件股份有限公司 | Intelligent management method for data slicing |
| CN117112267B (en) * | 2023-10-20 | 2024-01-23 | 成都华栖云科技有限公司 | Cache maintenance method of application interface |
| CN117785949B (en) * | 2024-02-28 | 2024-05-10 | 云南省地矿测绘院有限公司 | Data caching method, electronic equipment, storage medium and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455605A (en) * | 2013-09-04 | 2013-12-18 | 电子科技大学 | Deep searching method for files in internal network environment |
| CN103793485A (en) * | 2014-01-20 | 2014-05-14 | 锐达互动科技股份有限公司 | Method for inquiring network data by client terminal based on cached data |
| CN104168300A (en) * | 2013-05-17 | 2014-11-26 | 中国电信股份有限公司 | Content acceleration method and system |
| CN104468807A (en) * | 2014-12-12 | 2015-03-25 | 北京小渔无限信息技术有限公司 | Processing method, cloud end device, local devices and system for webpage cache |
| CN104767753A (en) * | 2015-04-08 | 2015-07-08 | 无锡天脉聚源传媒科技有限公司 | Method and device for processing message requests through server |
| CN106326389A (en) * | 2016-08-17 | 2017-01-11 | 深圳市金证科技股份有限公司 | Data-caching-based service request processing method and system |
| CN107203613A (en) * | 2017-05-23 | 2017-09-26 | 北京微影时代科技有限公司 | A kind of method and device of high concurrent request processing |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101266609B (en) * | 2008-04-30 | 2010-06-23 | 中山爱科数字科技有限公司 | Method for accomplishing medical data external inquiry for digital remote medical treatment |
| CN103530349A (en) * | 2013-09-30 | 2014-01-22 | 乐视致新电子科技（天津）有限公司 | Method and equipment for cache updating |
| CN106844784A (en) * | 2017-03-14 | 2017-06-13 | 上海网易小额贷款有限公司 | Data cache method, device and computer-readable recording medium |
-
2017
- 2017-10-18 CN CN201710969976.3A patent/CN109684358B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104168300A (en) * | 2013-05-17 | 2014-11-26 | 中国电信股份有限公司 | Content acceleration method and system |
| CN103455605A (en) * | 2013-09-04 | 2013-12-18 | 电子科技大学 | Deep searching method for files in internal network environment |
| CN103793485A (en) * | 2014-01-20 | 2014-05-14 | 锐达互动科技股份有限公司 | Method for inquiring network data by client terminal based on cached data |
| CN104468807A (en) * | 2014-12-12 | 2015-03-25 | 北京小渔无限信息技术有限公司 | Processing method, cloud end device, local devices and system for webpage cache |
| CN104767753A (en) * | 2015-04-08 | 2015-07-08 | 无锡天脉聚源传媒科技有限公司 | Method and device for processing message requests through server |
| CN106326389A (en) * | 2016-08-17 | 2017-01-11 | 深圳市金证科技股份有限公司 | Data-caching-based service request processing method and system |
| CN107203613A (en) * | 2017-05-23 | 2017-09-26 | 北京微影时代科技有限公司 | A kind of method and device of high concurrent request processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109684358A (en) | 2019-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684358B (en) | Data query method and device | |
| CN109947668B (en) | Method and device for storing data | |
| CN110598138B (en) | Processing method and device based on cache | |
| CN110545246A (en) | Token bucket-based current limiting method and device | |
| CN111614736A (en) | Network content resource scheduling method, domain name scheduling server and electronic equipment | |
| CN112445857A (en) | Resource quota management method and device based on database | |
| CN110321252B (en) | Skill service resource scheduling method and device | |
| CN109388655A (en) | A kind of method and apparatus of dynamic control of data access | |
| CN114612212A (en) | Business processing method, device and system based on risk control | |
| CN110071952B (en) | Service call quantity control method and device | |
| CN111831503A (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113783913A (en) | Message pushing management method and device | |
| CN111988403A (en) | Request processing method and system of electronic equipment, storage medium and electronic equipment | |
| CN114374657B (en) | Data processing method and device | |
| CN113779412B (en) | Message touch method, node and system based on blockchain network | |
| CN116185578A (en) | Scheduling method of computing task and executing method of computing task | |
| CN112688982B (en) | User request processing method and device | |
| CN115190125A (en) | Monitoring method and device for cache cluster | |
| CN112783914A (en) | Statement optimization method and device | |
| CN113329010B (en) | User access management method and system | |
| CN113114611A (en) | Method and device for managing blacklist | |
| CN113765969B (en) | Flow control method and device | |
| CN113760965B (en) | Data query method and device | |
| CN113778909B (en) | Method and device for caching data | |
| CN117478535B (en) | Log storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |