CN112597202B - Data query method and device, computer equipment and storage medium - Google Patents
Data query method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112597202B CN112597202B CN202011578005.4A CN202011578005A CN112597202B CN 112597202 B CN112597202 B CN 112597202B CN 202011578005 A CN202011578005 A CN 202011578005A CN 112597202 B CN112597202 B CN 112597202B
- Authority
- CN
- China
- Prior art keywords
- search engine
- server
- database
- query request
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data query method, a data query device, computer equipment and a storage medium. The method comprises the following steps: receiving a query request, and acquiring resource use information of a relational database server when the query request is a real-time query request; if the resource use information of the relational database server meets the preset conditions, calling the relational database server to acquire data corresponding to the query request from the relational database; if the resource use information of the relational database server does not accord with the preset condition, calling the search engine server to acquire data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database. By adopting the method, the database system can be switched at any time according to the resource use condition under the condition of high concurrency of the query request, and response overtime of the query request is avoided, so that the availability and the stability of the whole database system are improved.
Description
Technical Field
The present application relates to the field of database technologies, and in particular, to a data query method, apparatus, computer device, and storage medium.
Background
With the development of database technology, various types of databases have appeared, for example, a relational database, which is a database that uses a relational model to organize data, and stores data in rows and columns, and the storage form is easy to understand and convenient to use, and is a database that is currently used more frequently.
For a system of a relational database, a master-slave deployment mode is generally adopted, wherein a master database is used for providing a writing service, and a slave database is used for providing a query service. However, under the condition of high concurrency (peak of reading) of the query request, because the relational database adopts a row and column storage form, a plurality of copies have to be made to meet the query request, and in order to maintain the plurality of copies, the synchronization of data between the master and the slave consumes more system resources, so that the availability and stability of the whole database system are influenced under the high concurrency condition, and the response of the query request is overtime.
Disclosure of Invention
In view of the above, it is necessary to provide a data query method, apparatus, computer device and storage medium capable of improving the availability and stability of a database system in the case of high concurrency of query requests.
A method for data query, the method comprising:
receiving a query request, and acquiring resource use information of a relational database server when the query request is a real-time query request;
if the resource use information of the relational database server meets the preset conditions, calling the relational database server to acquire data corresponding to the query request from the relational database;
if the resource use information of the relational database server does not accord with the preset condition, calling the search engine server to acquire data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database.
In one embodiment, before receiving the query request, the method further includes: and acquiring data to be written, and calling the relational database server to write the data to be written into the relational database.
In one embodiment, before receiving the query request, the method further includes: and pushing the data to be written to the message queue so that the search engine server writes the data to be written into the search engine database when acquiring the data to be written in the message queue.
In one embodiment, after the relational database server is called to write the data to be written into the relational database, the method further includes: and extracting data from the relational database through the database middleware, and backing up the extracted data to the search engine database.
In one embodiment, before invoking the search engine server to obtain the data corresponding to the query request from the search engine database, the method further includes: judging whether the query request is a query request related to a core service; if not, the step of calling the search engine server to acquire the data corresponding to the query request from the search engine database is executed; if so, the step of calling the relational database server to acquire the data corresponding to the query request from the relational database is carried out.
In one embodiment, the search engine server includes a primary search engine server and a backup search engine server, the search engine database includes a primary search engine database and a backup search engine database, and the invoking search engine server obtains data corresponding to the query request from the search engine database includes: and acquiring heartbeat information of the main search engine server, judging whether the main search engine server is abnormal or not according to the heartbeat information of the main search engine server, and calling the standby search engine server to acquire data corresponding to the query request from the standby search engine database if the main search engine server is abnormal.
In one embodiment, the search engine server includes a primary search engine server and a backup search engine server, the search engine database includes a primary search engine database and a backup search engine database, and the invoking search engine server obtains data corresponding to the query request from the search engine database includes: and acquiring heartbeat information of the standby search engine server, judging whether the standby search engine server is abnormal or not according to the heartbeat information of the standby search engine server, and calling the main search engine server to acquire data corresponding to the query request from the main search engine database if the standby search engine server is abnormal.
In one embodiment, the method further comprises: if the main search engine server and the standby search engine server are not abnormal, the currently available search engine server is called to perform data query processing according to the resource use information of the main search engine server and the standby search engine server and the preset resource division proportion of the main search engine server and the standby search engine server.
In one embodiment, the method further comprises: and when the query request is an asynchronous query request, calling a search engine server to acquire data corresponding to the query request from a search engine database.
A data query device, said device comprising:
the information acquisition module is used for receiving the query request and acquiring the resource use information of the relational database server when the query request is a real-time query request;
the first calling module is used for calling the relational database server to acquire data corresponding to the query request from the relational database if the resource use information of the relational database server meets the preset condition;
the second calling module is used for calling the search engine server to acquire data corresponding to the query request from the search engine database if the resource use information of the relational database server does not accord with the preset condition; wherein the same data is written into the relational database earlier than the search engine database.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the data query method described above when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the data query method described above.
According to the data query method, the data query device, the computer equipment and the storage medium, when a real-time query request is received, whether the relational database server with higher real-time performance is called for processing or not is judged, the availability of the relational database with higher real-time performance can be preferentially ensured, when the relational database server exceeds load, the search engine server with lower real-time performance is called for performing degradation processing on the query request, the problem that the database system is unavailable or unstable due to overload of the relational database server can be avoided, meanwhile, the database system can be switched at any time according to resource conditions under the condition of high concurrency, response overtime of the query request is avoided, and the availability of the whole database system is improved.
Drawings
FIG. 1 is a diagram of an application environment of a data query method in one embodiment;
FIG. 2 is a flow diagram that illustrates a methodology for querying data in one embodiment;
FIG. 3 is a flow diagram that illustrates the steps of writing data to a database, in one embodiment;
FIG. 4 is a technical architecture diagram of a data query method based on a hierarchically deployed data storage structure in an application example;
FIG. 5 is a diagram of a technical architecture for performing data query based on a real-time query request in an application example;
FIG. 6 is a diagram of a technical architecture for performing data queries based on asynchronous query requests in an application example;
FIG. 7 is a diagram of a technical architecture for writing data to a relational database according to a service request in an application example;
FIG. 8 is a block diagram showing the structure of a data search device according to an embodiment;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It is to be understood that the term "and/or", as used herein, describes an associative relationship of associated objects, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The data query method provided by the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The server 104 receives the query request sent by the terminal 102, when the query request is a real-time query request, the resource usage information of the relational database server 106 is obtained, if the resource usage information of the relational database server meets a preset condition, the relational database server is called to obtain data corresponding to the query request from the relational database, if the resource usage information of the relational database server 106 does not meet the preset condition, the search engine server 108 is called to obtain the data corresponding to the query request from the search engine database, and the time for writing the same data into the relational database is earlier than the time for writing the same data into the search engine database.
The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server can be implemented by an independent server or a server cluster formed by a plurality of servers. Server 106 may replace relational database server 106 or may be any one of a cluster of relational database servers 106.
In one embodiment, as shown in fig. 2, a data query method is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:
step S202: and receiving a query request, and acquiring the resource use information of the relational database server when the query request is a real-time query request.
The real-time query request refers to a query request with a high requirement on timeliness. The relational database is a database which is built on the basis of the relational database model and processes data in the database by using concepts and methods such as set algebra and the like. The relational database server refers to a server or a server cluster loaded with a relational database management application program, and is used for providing data writing or query service based on the relational database. The resource usage information is information representing resource usage, and may specifically include CPU usage information, memory usage information, IO status information, soft interrupt status information, network card bandwidth status information, and the like.
Specifically, when the server receives a query request of data sent by the consumption terminal, it may first determine whether the query request is a query request with a high requirement on timeliness, that is, whether the query request is a real-time query request, and if so, may obtain resource usage information of the relational database server or the server cluster, and analyze a current load condition, a resource utilization condition, and the like of the relational database server through the resource usage information.
Step S204: and if the resource use information of the relational database server meets the preset conditions, calling the relational database server to acquire data corresponding to the query request from the relational database.
Specifically, when the resource usage information of the relational database server meets a preset condition, for example, if the performance of the current relational database server meets a preset condition that the representation performance is good, that is, when the available resource of the relational database system reaches a preset threshold, the relational database server may be preferentially called to obtain the data corresponding to the query request from the relational database. The reason why the relational database server is preferentially called to perform query processing is that the data in the relational database has high timeliness, for example, the data in the relational database can be written in a real-time writing manner, so that the response efficiency of real-time query can be ensured.
Step S206: if the resource use information of the relational database server does not accord with the preset condition, calling the search engine server to acquire data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database.
The data in the search engine database has lower timeliness than the data in the relational database, that is, the time for writing the same data into the relational database is earlier than the time for writing the same data into the search engine database. The Search engine server may be an ES (Elastic Search) engine server or the like.
Specifically, when the performance problem of the relational database system is analyzed according to the resource use information of the relational database server, the relational database server is degraded and switched to call the search engine server to perform query processing on the current query request in due time, so that the pressure of the relational database system can be reduced, the availability of the search engine database can be improved on the premise of ensuring most query timeliness, and the stability and the availability of the whole database system can be ensured.
According to the data query method, when a real-time query request is received, whether the relational database server with higher real-time performance is called or not is judged, the availability of the relational database with higher real-time performance can be preferentially ensured, when the relational database server exceeds the load, the search engine server with lower real-time performance is called to perform degradation query on the query request, the problem that the database system is unavailable or unstable due to overload of the relational database server is avoided, meanwhile, the database can be switched at any time according to the resource condition under the high concurrency condition, the response overtime of the query request is avoided, and the availability of the whole database system is improved.
In one embodiment, as shown in FIG. 3, FIG. 3 illustrates a flow diagram of the steps of writing data to a database in one embodiment. Before receiving the query request, the method further includes a step of writing data into the database, and specifically includes: step S302: and acquiring data to be written. Step S304: and calling the relational database server to write the data to be written into the relational database.
Specifically, after receiving a service processing request sent by a client terminal, processing service data according to the service processing request, and obtaining data to be written to a database after the processing is completed. After the server acquires the data to be written, the server can directly call the relational database server to write the data to be written into the relational database in real time, so that the timeliness of data query of the relational database data is ensured.
In one embodiment, prior to receiving the query request, the step of writing data to the database comprises: step S302: and acquiring data to be written. Step S306: and pushing the data to be written to the message queue so that the search engine server writes the data to be written into the search engine database when acquiring the data to be written in the message queue.
Specifically, when the server acquires the data to be written, on one hand, the data to be written can be directly written into the relational database by calling the relational database, and on the other hand, the data to be written can be pushed into the message queue for consumption by other consumption terminals (search engine servers). The search engine server can periodically access the message queue through a timing task and other modes, acquire data from the message queue, and write the data to be written into the search engine database when acquiring the data to be written.
In the two embodiments, when the client terminal requests to write data, the data to be written is written into the database through two paths, that is, the data is written into the relational database in real time, and the data is backed up to the search engine database through another path, so that data synchronization is not required to be performed by deploying the master-slave relational database while data loss is avoided and data security is ensured, and therefore, resources of a relational database system are not occupied, the performance of real-time query of the relational database can be improved, and the availability of the relational database under high concurrence of query requests is further improved.
In one embodiment, after the relational database server is called to write the data to be written into the relational database, the step S308 may be included: and extracting data from the relational database through the database middleware, and backing up the extracted data to the search engine database.
In this embodiment, the data may also be extracted from the relational database by using the database middleware to implement backup of the data, for example, data extraction and backup may be performed in a manner of database Binlog (binary log) subscription, and the like. The data backup is carried out by adopting the principle of a copy state machine, and the backup complete data can be acquired as required to prevent the problems caused by data loss or time pollution of a real-time channel.
Preferably, the search engine database may include a main search engine database and a standby search engine database, and when data is written, and the search engine server obtains data to be written in the message queue, the data to be written is written in the search engine database, including: and when the search engine server acquires the data to be written in the message queue, writing the data to be written in the main search engine database. Backing up the extracted data to a search engine database, comprising: and backing up the extracted data to a standby search engine database.
In one embodiment, before invoking the search engine server to obtain the data corresponding to the query request from the search engine database, the method further includes: judging whether the query request is a query request related to a core service; if not, the step of calling the search engine server to acquire the data corresponding to the query request from the search engine database is executed; if so, the step of calling the relational database server to acquire the data corresponding to the query request from the relational database is carried out.
In this embodiment, for the real-time query request, a suitable database server is called in a layer-by-layer degradation manner to perform query processing according to the current performance of the database and the service level related to the query request. When the relational database system has performance problems, the non-core service can be degraded to the search engine database system, but the processing of the relational database system on the core service is maintained, and the influence of the non-core service on the core service is reduced, so that the timeliness of the core service query is ensured, and the stability of the whole database system is improved. Further, flexible transactions may be employed to ensure basic availability of primary and backup search engine database functions and eventual consistency of data.
In one embodiment, the search engine server includes a primary search engine server and a backup search engine server, the search engine database includes a primary search engine database and a backup search engine database, and the invoking search engine server obtains data corresponding to the query request from the search engine database includes: and acquiring heartbeat information of the main search engine server, judging whether the main search engine server is abnormal or not according to the heartbeat information of the main search engine server, and calling the standby search engine server to acquire data corresponding to the query request from the standby search engine database if the main search engine server is abnormal.
In one embodiment, invoking the search engine server to obtain data corresponding to the query request from the search engine database includes: and acquiring heartbeat information of the standby search engine server, judging whether the standby search engine server is abnormal or not according to the heartbeat information of the standby search engine server, and calling the main search engine server to acquire data corresponding to the query request from the main search engine database if the standby search engine server is abnormal.
In the two embodiments, the running states of the main and standby servers can be judged through the heartbeat information of the main and standby servers of the search engine database, and when the main search engine server is abnormal, resources can be completely divided into the standby search engine servers so as to ensure that the standby search engine servers can provide data query services; when the standby search engine server is abnormal, resources can be completely divided into the main search engine server to ensure that the main search engine server can provide data query service, namely at least one search engine server or a group of search engine server clusters can provide query service to realize timely transfer of faults.
Further, the main server and the standby server can also adopt an extension room deployment mode so as to achieve the purpose that the standby server or the server cluster can also normally provide service even if the machine room of the main server or the server cluster is powered off.
It is worth noting here that there is little possibility of simultaneous failure of both primary and secondary search engine servers (e.g., ES clusters). Since, the ES cluster itself has three characteristics to guarantee availability of the basic cluster: (1) the cluster state is synchronized and a specified number of replies are received before being considered successful. (2) When the node joins or leaves, the non-master node fails, the data can be recovered by performing the transcog when the non-master node recovers, the master node leaves, and other slave nodes can reselect the master node in an election mode. (3) The fragment copies are synchronous, similar to the master-slave architecture of a relational database, the node where the master fragment is positioned has problems, the node where the slave fragment is positioned can replace the original service, and formally, the master fragment and the slave fragment are not always positioned on one machine.
In one embodiment, the method further comprises: if the main search engine server and the standby search engine server are not abnormal, the currently available search engine server is called to perform data query according to the resource use information of the main search engine server and the standby search engine server and the preset resource division proportion of the main search engine server and the standby search engine server.
In this embodiment, when neither the main search engine server nor the backup search engine server is abnormal, resources may be divided for the main and backup search engine servers according to a preset resource division ratio, and generally, the resource flow of the default backup search engine server is smaller than that of the main search engine server. And further, the current available search engine server can be selected and called to perform query processing by combining the current resource use information of the main and standby search engine servers.
In one embodiment, the method further comprises: and when the query request is an asynchronous query request, calling a search engine server to acquire data corresponding to the query request from a search engine database.
In this embodiment, when the real-time requirement of the consumer on the data query is not so high, for example, when the query request is an asynchronous query request, the search engine server may be invoked to provide the query service, and by degrading the asynchronous query request with lower real-time requirement to the search engine database system to provide the query service, the pressure of the relational database system may be reduced, and the response efficiency of the real-time query request may be improved.
The following describes the data query method according to the present application in detail with reference to an application example. As shown in fig. 4, fig. 4 is a technical architecture diagram illustrating a data query method based on a hierarchically deployed data storage structure in an application instance.
The storage for data can be hierarchically deployed in the following manner:
1. when the server receives the data requested to be written by the client, the data can be directly written into the relational database.
2. When the server receives the data which is sent by the client and requested to be written, the data can be pushed to the message queue, and the data is written into the main search engine database through real-time consumption of the message queue.
3. The server side can acquire the complete data backup of the relational database to the standby search engine database in a database middleware extraction mode.
The query for data may be implemented based on the deployed data storage structure:
1. after receiving the data writing request, the data is directly written into the relational database, so that the data timeliness of the relational database is highest, and when the data is queried in real time, the real-time query route 1 can be preferentially adopted to acquire the data from the relational database in real time.
2. Because the data of the main search engine database is written in by means of consuming the message queue, the timeliness of the data of the main search engine database is lower than that of the relational database, so that the real-time query route 2 can be used as a degraded query route of the real-time query route 1, when the relational database has performance problems, the non-core service can be degraded to the real-time query route 2 for query, and the influence of the non-core service on the core service is reduced. Or, when the relational database fails and is not available, the core service can be degraded to the real-time query line 2, so as to ensure that the core service is not interrupted.
3. The standby search engine database is used as a later layer of the multi-layer data storage, the replication state machine principle is adopted to ensure that complete data can be obtained at any time to prevent the problem of message loss or time pollution of a real-time channel, and query service can be provided if necessary.
4. When the real-time requirement of the consumption end (asynchronous query end) on the data is not high, the query service can be accessed to the asynchronous query line 1, and the query service is provided through the main search engine server, so that the overall service capability of the service end can be improved, and the system pressure of the relational database can be reduced. In addition, the asynchronous query circuit 2 can also be used as a degraded route of the asynchronous query circuit 1 or the real-time query circuit 2 while providing a complete data backup, so as to improve the availability of the whole database system.
Specifically, referring to fig. 5, fig. 5 is a schematic diagram illustrating a technical architecture for performing data query according to a real-time query request in an application example, which is described in detail as follows:
1. when the resource use condition of the relational database triggers an alarm, the alarm information is timely reported to the server, and the server stores the information into the distributed cache.
2. When a real-time query request comes, if the resources of the existing relational database are good and no load is generated, the request is transferred to a relational database server or a server cluster for real-time query.
3. Otherwise, performing degradation processing, and switching to the search engine database to perform lossy query, where the loss is caused by that the search engine database adopts a timing submission policy in order to improve the writing performance, and secondly, the search engine database lacks transaction management, and although a local message table solution can be introduced, these all cause data to be slightly delayed, but are still friendly with respect to fusing some scenes.
Referring to fig. 6, fig. 6 is a schematic diagram illustrating a technical architecture for performing data query according to an asynchronous query request in an application example, which is described in detail as follows:
1. the server sends a heartbeat information request to the main and standby search engine server clusters at regular time, and stores the obtained result in a cache.
2. When the query request of the peripheral system comes (the query request of the default peripheral system is an asynchronous query request), if the heartbeats of the main and standby clusters are normal, the main and standby clusters are shunted according to the resource condition, and the flow of the default standby cluster is little or almost no.
3. When the main cluster is abnormal, all the traffic is divided into the standby clusters, and when the standby clusters are abnormal, all the traffic is divided into the main cluster, so that at least one server or one group of server clusters can provide query service.
Referring to fig. 7, fig. 7 is a diagram illustrating a technical architecture for writing data into a relational database according to a service request in an application example. Especially when high concurrency issues involving writes are involved, the solution idea as shown in fig. 7 can be employed:
when a service request comes, the number of the requests is controlled in a funnel flow control manner, for example, if the service request is within a threshold range, normal consumption is performed, a part exceeding the threshold is managed by a cache queue (redis), and a new thread polls the part exceeding the threshold at regular time. After the data processing is performed according to the service request, the data is written into the database, for example, the data may be written in a form of sub-database and sub-table. And recording the abnormal data to an abnormal table when the abnormal data occurs, and processing the abnormal data by the scheduling platform at regular time.
Several concepts introduced here are explained as follows:
funnel flow control: the algorithm takes a funnel as a prototype, the sand flowing out in a specific time is fixed, the flow rate is also fixed, and the core variable is as follows: capacity, rate, last trigger time, remaining capacity.
Database and table division: in order to solve the problem of high concurrency, data fragmentation can be considered and put into a plurality of database clusters for management, the problem of table amount can be solved, and data can be subjected to table division.
It should be understood that although the various steps in the flow charts of fig. 2-3 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-3 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 8, there is provided a data query apparatus including: the information obtaining module 10, the first calling module 20 and the second calling module 30, wherein:
the information acquisition module 10 is configured to receive a query request, and when the query request is a real-time query request, acquire resource usage information of the relational database server;
the first calling module 20 is configured to, if the resource usage information of the relational database server meets a preset condition, call the relational database server to obtain data corresponding to the query request from the relational database;
the second calling module 30 is configured to, if the resource usage information of the relational database server does not meet the preset condition, call the search engine server to obtain data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database.
In an embodiment, the apparatus further includes a data writing module 40, where the data writing module 40 is configured to obtain data to be written, and call the relational database server to write the data to be written into the relational database.
In one embodiment, the data writing module 40 pushes the data to be written to the message queue, so that the search engine server writes the data to be written to the search engine database when acquiring the data to be written in the message queue.
In one embodiment, the data writing module 40 is further configured to extract data from the relational database through the database middleware, and backup the extracted data to the search engine database.
In one embodiment, the first invoking module 20 is further configured to determine whether the query request is a query request related to a core service before invoking the search engine server to obtain data corresponding to the query request from the search engine database; if not, the step of calling the search engine server to acquire the data corresponding to the query request from the search engine database is executed; if so, the step of calling the relational database server to acquire the data corresponding to the query request from the relational database is carried out.
In one embodiment, the search engine server includes a main search engine server and a backup search engine server, the search engine database includes a main search engine database and a backup search engine database, the second invoking module 30 obtains heartbeat information of the main search engine server, determines whether the main search engine server is abnormal according to the heartbeat information of the main search engine server, and invokes the backup search engine server to obtain data corresponding to the query request from the backup search engine database if the main search engine server is abnormal.
In one embodiment, the search engine server includes a main search engine server and a backup search engine server, the search engine database includes a main search engine database and a backup search engine database, the second invoking module 30 obtains heartbeat information of the backup search engine server, determines whether the backup search engine server is abnormal according to the heartbeat information of the backup search engine server, and invokes the main search engine server to obtain data corresponding to the query request from the main search engine database if the backup search engine server is abnormal.
In one embodiment, the second invoking module 30 is further configured to invoke a currently available search engine server to perform data query processing according to the resource usage information of the main search engine server and the backup search engine server and a preset resource dividing ratio of the main search engine server and the backup search engine server if neither the main search engine server nor the backup search engine server is abnormal.
In one embodiment, the second calling module 30 is further configured to, when the query request is an asynchronous query request, enter a step of calling the search engine server to obtain data corresponding to the query request from the search engine database.
For the specific definition of the data query device, reference may be made to the above definition of the data query method, which is not described herein again. The modules in the data query device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device comprises a processor, a memory and a network interface which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data query method.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: receiving a query request, and acquiring resource use information of a relational database server when the query request is a real-time query request; if the resource use information of the relational database server meets the preset conditions, calling the relational database server to acquire data corresponding to the query request from the relational database; if the resource use information of the relational database server does not accord with the preset condition, calling the search engine server to acquire data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database.
In one embodiment, prior to receiving the query request, the processor executing the computer program further performs obtaining data to be written, invoking the relational database server to write the data to be written to the relational database.
In one embodiment, prior to receiving the query request, the processor executing the computer program further enables pushing data to be written to the message queue, so that the search engine server writes the data to be written to the search engine database when acquiring the data to be written in the message queue.
In one embodiment, execution of the computer program by the processor further effects invoking the relational database server to write the data to be written to the relational database, and thereafter, further effects extracting data from the relational database via the database middleware and backing up the extracted data to the search engine database.
In one embodiment, the processor executing the computer program further enables determining whether the query request is a query request relating to a core service before invoking the search engine server to obtain data corresponding to the query request from the search engine database; if not, the step of calling the search engine server to acquire the data corresponding to the query request from the search engine database is executed; if so, the step of calling the relational database server to acquire the data corresponding to the query request from the relational database is carried out.
In one embodiment, the processor executes a computer program to realize that when the search engine server is called to obtain data corresponding to the query request from the search engine database, the heartbeat information of the main search engine server is obtained, whether the main search engine server is abnormal or not is judged according to the heartbeat information of the main search engine server, and if the main search engine server is abnormal, the standby search engine server is called to obtain the data corresponding to the query request from the standby search engine database.
In one embodiment, the processor executes a computer program to call the search engine server to obtain data corresponding to the query request from the search engine database, specifically, to obtain heartbeat information of the backup search engine server, determine whether the backup search engine server is abnormal according to the heartbeat information of the backup search engine server, and if so, call the main search engine server to obtain data corresponding to the query request from the main search engine database.
In one embodiment, the processor executes the computer program to further implement that if no abnormality occurs in either the main search engine server or the backup search engine server, the currently available search engine server is called to perform data query processing according to the resource usage information of the main search engine server and the backup search engine server and the preset resource division ratio of the main search engine server and the backup search engine server.
In one embodiment, the processor executing the computer program further performs the step of invoking the search engine server to obtain data corresponding to the query request from the search engine database when the query request is an asynchronous query request.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: receiving a query request, and acquiring resource use information of a relational database server when the query request is a real-time query request; if the resource use information of the relational database server meets the preset conditions, calling the relational database server to acquire data corresponding to the query request from the relational database; if the resource use information of the relational database server does not accord with the preset condition, calling the search engine server to acquire data corresponding to the query request from the search engine database; wherein the same data is written into the relational database earlier than the search engine database.
In one embodiment, prior to receiving the query request, execution of the computer program by the processor further effects obtaining data to be written, invoking the relational database server to write the data to be written to the relational database.
In one embodiment, the computer program executed by the processor further enables pushing data to be written to the message queue prior to receiving the query request, so that the search engine server writes the data to be written to the search engine database when acquiring the data to be written in the message queue.
In one embodiment, the computer program executed by the processor further enables, after invoking the relational database server to write the data to be written into the relational database, extracting the data from the relational database through the database middleware and backing up the extracted data to the search engine database.
In one embodiment, the computer program executed by the processor further enables determining whether the query request is a query request related to a core service before invoking the search engine server to obtain data corresponding to the query request from the search engine database; if not, the step of calling the search engine server to acquire the data corresponding to the query request from the search engine database is executed; if so, the step of calling the relational database server to acquire the data corresponding to the query request from the relational database is carried out.
In one embodiment, the computer program is executed by the processor to realize that when the search engine server is called to obtain data corresponding to the query request from the search engine database, the heartbeat information of the main search engine server is obtained, whether the main search engine server is abnormal or not is judged according to the heartbeat information of the main search engine server, and if the main search engine server is abnormal, the backup search engine server is called to obtain the data corresponding to the query request from the backup search engine database.
In one embodiment, the computer program is executed by the processor to implement invoking the search engine server to obtain data corresponding to the query request from the search engine database, specifically implement obtaining heartbeat information of the backup search engine server, determine whether the backup search engine server is abnormal according to the heartbeat information of the backup search engine server, and if so, invoke the main search engine server to obtain data corresponding to the query request from the main search engine database.
In one embodiment, the computer program executed by the processor further enables invoking a currently available search engine server for data query processing based on resource usage information of the primary search engine server and the backup search engine server and a preset resource partition ratio of the primary search engine server and the backup search engine server if neither the primary search engine server nor the backup search engine server is abnormal.
In one embodiment, the computer program executed by the processor further performs the step of invoking the search engine server to obtain data corresponding to the query request from the search engine database when the query request is an asynchronous query request.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data query, the method comprising:
receiving a query request, and acquiring resource use information of a relational database server when the query request is a real-time query request;
if the resource use information of the relational database server meets a preset condition, calling the relational database server to acquire data corresponding to the query request from a relational database;
if the resource use information of the relational database server does not accord with the preset condition, calling a search engine server to acquire data corresponding to the query request from a search engine database; and writing the same data into the relational database earlier than the search engine database.
2. The method of claim 1, wherein prior to said receiving a query request, the method further comprises:
acquiring data to be written, and calling the relational database server to write the data to be written into the relational database;
and/or
And pushing the data to be written to a message queue so that the search engine server writes the data to be written into the search engine database when acquiring the data to be written in the message queue.
3. The method of claim 2, wherein after the invoking the relational database server to write the data to be written to the relational database, the method further comprises:
and extracting data from the relational database through database middleware, and backing up the extracted data to the search engine database.
4. The method of claim 1, wherein before the invoking the search engine server obtains the data corresponding to the query request from a search engine database, the method further comprises:
judging whether the query request is a query request related to a core service;
if not, executing the step of acquiring the data corresponding to the query request from a search engine database by calling the search engine server;
and if so, acquiring the data corresponding to the query request from the relational database by calling the relational database server.
5. The method of claim 1, wherein the search engine server comprises a primary search engine server and a backup search engine server, the search engine database comprises a primary search engine database and a backup search engine database, and the invoking search engine server obtains data corresponding to the query request from the search engine database comprises:
acquiring heartbeat information of the main search engine server, judging whether the main search engine server is abnormal or not according to the heartbeat information of the main search engine server, and calling the standby search engine server to acquire data corresponding to the query request from the standby search engine database if the main search engine server is abnormal; or the like, or, alternatively,
and acquiring heartbeat information of the standby search engine server, judging whether the standby search engine server is abnormal or not according to the heartbeat information of the standby search engine server, and calling the main search engine server to acquire data corresponding to the query request from the main search engine database if the standby search engine server is abnormal.
6. The method of claim 5, further comprising:
if the main search engine server and the standby search engine server are not abnormal, calling the currently available search engine server to perform data query processing according to the resource use information of the main search engine server and the standby search engine server and the preset resource division ratio of the main search engine server and the standby search engine server.
7. The method according to any one of claims 1 to 6, further comprising:
and when the query request is an asynchronous query request, the step of calling the search engine server to acquire the data corresponding to the query request from a search engine database is carried out.
8. A data query apparatus, characterized in that the apparatus comprises:
the information acquisition module is used for receiving a query request and acquiring resource use information of the relational database server when the query request is a real-time query request;
the first calling module is used for calling the relational database server to acquire data corresponding to the query request from the relational database if the resource use information of the relational database server meets a preset condition;
the second calling module is used for calling the search engine server to acquire data corresponding to the query request from the search engine database if the resource use information of the relational database server does not accord with the preset condition; wherein the time of writing the same data into the relational database is earlier than the time of writing the same data into the search engine database.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011578005.4A CN112597202B (en) | 2020-12-28 | 2020-12-28 | Data query method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011578005.4A CN112597202B (en) | 2020-12-28 | 2020-12-28 | Data query method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112597202A CN112597202A (en) | 2021-04-02 |
| CN112597202B true CN112597202B (en) | 2022-08-05 |
Family
ID=75202819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011578005.4A Active CN112597202B (en) | 2020-12-28 | 2020-12-28 | Data query method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112597202B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220722A (en) * | 2021-04-26 | 2021-08-06 | 深圳市云网万店科技有限公司 | Data query method and device, computer equipment and storage medium |
| CN113377866A (en) * | 2021-06-10 | 2021-09-10 | 全球能源互联网研究院有限公司 | Load balancing method and device for virtualized database proxy service |
| CN113568907A (en) * | 2021-07-14 | 2021-10-29 | 彩讯科技股份有限公司 | Method, device, server and storage medium for processing station information message |
| CN116821192B (en) * | 2023-08-30 | 2023-11-24 | 中国中金财富证券有限公司 | Asset data query method and device and computer equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968679A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Data query method and device |
| CN111694866A (en) * | 2020-06-09 | 2020-09-22 | 南京领行科技股份有限公司 | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium |
| CN112100179A (en) * | 2020-09-11 | 2020-12-18 | 北京明略昭辉科技有限公司 | HBASE-based data fusion method, HBASE-based data fusion device, HBASE-based data fusion equipment and computer readable medium |
-
2020
- 2020-12-28 CN CN202011578005.4A patent/CN112597202B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112597202A (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112597202B (en) | Data query method and device, computer equipment and storage medium | |
| US20200257593A1 (en) | Storage cluster configuration change method, storage cluster, and computer system | |
| JP6404907B2 (en) | Efficient read replica | |
| US11907561B2 (en) | Data backup method and apparatus | |
| CN102411639B (en) | Multi-copy storage management method and system of metadata | |
| CN103763155A (en) | Multi-service heartbeat monitoring method for distributed type cloud storage system | |
| JP2007241486A (en) | Memory system | |
| CN111061431B (en) | Distributed storage method, server and client | |
| CN114900532A (en) | Power data disaster tolerance method, system, device, computer equipment and storage medium | |
| CN110377664B (en) | Data synchronization method, device, server and storage medium | |
| CN111176584A (en) | Data processing method and device based on hybrid memory | |
| CN107943615B (en) | Data processing method and system based on distributed cluster | |
| CN113946491A (en) | Microservice data processing method, microservice data processing device, computer equipment and storage medium | |
| CN108763312A (en) | A kind of slave back end screening technique based on load | |
| Mookdarsanit et al. | Light-weight operation of a failover system for Cloud computing | |
| CN114564340A (en) | Distributed software high-availability method for aerospace ground system | |
| CN113536034A (en) | Data writing method and data reading method based on bloom filter | |
| CN112711606A (en) | Database access method and device, computer equipment and storage medium | |
| CN103685359A (en) | Data processing method and device | |
| CN117478299B (en) | Block chain consensus algorithm switching method, device and computer equipment | |
| CN115730016B (en) | Data synchronization method, system, device, computer equipment and storage medium | |
| CN112801769B (en) | Accounting data processing method, device and system | |
| CN115334000B (en) | Data transmission method, device, equipment and medium based on multi-control storage system | |
| CN112559445B (en) | Data writing method and device | |
| CN113485978B (en) | Method, system and memory for improving read-write throughput capacity of file storage NAS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |