CN115357641A

CN115357641A - Data query method and device, electronic equipment and storage medium

Info

Publication number: CN115357641A
Application number: CN202210998447.7A
Authority: CN
Inventors: 余志佳; 马文超; 隆盼
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-08-19
Filing date: 2022-08-19
Publication date: 2022-11-18

Abstract

The disclosure provides a data query method, a data query device, electronic equipment and a storage medium, which relate to the technical fields of cloud computing, cloud storage, big data and the like, and the specific implementation scheme is as follows: inquiring a plurality of indexes matched with the inquiry conditions from a distributed index library, and writing the indexes into a distributed message queue in a distributed cluster; acquiring a first index without a mark from a distributed message queue, and inquiring first service data matched with the first index from a distributed database; and marking a first index in the distributed message queue through the distributed cluster in response to the first service data being queried. Therefore, the situation that the service data is repeatedly inquired and acquired due to the fact that the marked indexes are acquired from the distributed message queue when the equipment is in downtime or the instance is migrated can be avoided, and therefore the data inquiring efficiency can be improved, and the resource occupation can be reduced.

Description

Data query method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the technical field of artificial intelligence, and in particular, to the technical fields of cloud computing, cloud storage, big data, and the like, and in particular, to a data query method, apparatus, electronic device, and storage medium.

Background

With the continuous development of internet technology and information technology, business data generated by enterprises is increasingly expanded. The massive service data may be stored in a distributed database, for example, structured service data (such as service data in the form of key-value pairs (KV) may be stored in the distributed database, which may also be referred to as a KV library.

In order to improve the query efficiency of business data, it is very important how to realize fast query of business data from a distributed database.

Disclosure of Invention

The disclosure provides a data query method, a data query device, an electronic device and a storage medium.

According to an aspect of the present disclosure, there is provided a data query method including:

acquiring a query condition, and querying a plurality of indexes matched with the query condition from a distributed index library; the distributed index database is used for storing indexes corresponding to all service data in the distributed database;

writing the plurality of indexes to a distributed message queue in a distributed cluster;

acquiring a first index without a mark from the distributed message queue, and inquiring first service data matched with the first index from the distributed database;

marking the first index in the distributed message queue by the distributed cluster in response to querying the first service data.

According to another aspect of the present disclosure, there is provided a data query apparatus including:

the query module is used for acquiring query conditions and querying a plurality of indexes matched with the query conditions from a distributed index library; the distributed index database is used for storing indexes corresponding to all service data in the distributed database;

a write module for writing the plurality of indices to a distributed message queue in a distributed cluster;

the acquisition module is used for acquiring a first unmarked index from the distributed message queue and inquiring first service data matched with the first index from the distributed database;

and the sending module is used for responding to the first service data inquired, and marking the first index in the distributed message queue through the distributed cluster.

According to still another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform a data query method set forth in the above-described aspect of the disclosure.

According to still another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium of computer instructions for causing a computer to perform the data query method set forth in the above aspect of the present disclosure.

According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the data query method set forth in the above aspect of the present disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a schematic diagram illustrating a data query and transmission flow in the related art;

fig. 2 is a schematic flowchart of a data query method according to a first embodiment of the disclosure;

fig. 3 is a schematic flowchart of a data query method according to a second embodiment of the disclosure;

fig. 4 is a schematic flowchart of a data query method according to a third embodiment of the present disclosure;

fig. 5 is a schematic flowchart of a data query method according to a fourth embodiment of the disclosure;

fig. 6 is a schematic flowchart of a data query method according to a fifth embodiment of the present disclosure;

FIG. 7 is a schematic diagram of an implementation of any embodiment of the disclosure;

fig. 8 is a schematic structural diagram of a data query apparatus according to a sixth embodiment of the present disclosure;

FIG. 9 illustrates a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

At present, for a KV-type distributed database (Data Base, abbreviated as DB), service Data stored in the distributed database can be queried through rowkey of each service Data. Specifically, as shown in fig. 1, when scanning each page of data in the distributed database, the query mode of the service data may be to scan the page of data according to the rowkey identifier returned by the previous page of data.

However, the above method can only acquire data by a single thread, each data in the distributed database can only be scanned sequentially, the data query efficiency is low, and the query rate per second (quieries-per-second) of data issue cannot be improved.

Moreover, in the data scanning process, if instance migration or machine downtime occurs, data transmission is interrupted, a retry mechanism needs to be executed, data is scanned from beginning to end again, and the scanned data is sent to other resource parties, which is relatively inefficient.

Namely, the following problems exist in fig. 1:

firstly, in the process of acquiring massive service data from a distributed database and transmitting the acquired service data from one service department to another service department, the QPS is uncontrollable and negligibly small;

secondly, when the data volume of the service data to be queried is large, not only is the data transmission time consumed long, but also if the machine responsible for transmission migrates or crashes, all the service data needs to be retransmitted, which results in low transmission efficiency of the service data, repetition of the transmitted service data, and resource waste.

Therefore, in view of at least one of the problems mentioned above, the present disclosure provides a data query method, apparatus, electronic device and storage medium.

A data query method, an apparatus, an electronic device, and a storage medium according to embodiments of the present disclosure are described below with reference to the accompanying drawings.

Fig. 2 is a schematic flow chart of a data query method according to a first embodiment of the disclosure.

The data query method is exemplified by being configured in a data query device, and the data query device can be applied to any electronic equipment, so that the electronic equipment can execute a data query function.

The electronic device may be any device with computing capability, for example, a computer, a mobile terminal, a server, and the like, and the mobile terminal may be, for example, a hardware device with various operating systems, touch screens, and/or display screens, such as an in-vehicle device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and the like.

As shown in fig. 2, the data query method may include the steps of:

step 201, obtaining a query condition, and querying a plurality of indexes matched with the query condition from a distributed index library; the distributed index database is used for storing indexes corresponding to all service data in the distributed database.

In the embodiment of the present disclosure, the query condition may be provided for the relevant person, for example, the query condition may be used to indicate which time interval to query the generated service data, or to indicate which service to query the service data, or to indicate which field to query the service data, and the like, which is not limited by the present disclosure.

In the embodiment of the present disclosure, according to a query condition, a plurality of indexes matching with the query condition may be queried from the distributed index library. The distributed index database is used for storing indexes corresponding to all service data in the distributed database.

As an example, the distributed database is a KV-type database (KV library for short), that is, the index of the service data may be rowkey.

Step 202, writing a plurality of indexes into a distributed message queue in a distributed cluster.

In embodiments of the present disclosure, a plurality of indices that match a query condition may be written to a distributed message queue in a distributed cluster.

And 203, acquiring a first unmarked index from the distributed message queue, and acquiring first service data matched with the first index from a distributed database.

In the embodiment of the present disclosure, an unmarked index (denoted as a first index in the present disclosure) may be obtained from the distributed message queue, and service data (denoted as first service data in the present disclosure) matching the first index may be queried from the distributed database.

Step 204, in response to querying the first service data, marking a first index in the distributed message queue by the distributed cluster.

In this embodiment of the present disclosure, in a case of querying or acquiring the first service data, the first index in the distributed message queue may be marked by the distributed cluster. For example, an acknowledgement response may be sent to a service node where a distributed message queue in the distributed cluster is located, and accordingly, after receiving the acknowledgement response, the service node may mark the first index in the distributed message queue.

The data query method of the embodiment of the disclosure queries a plurality of indexes matched with query conditions from a distributed index database, and writes the plurality of indexes into a distributed message queue in a distributed cluster; acquiring a first index without a mark from a distributed message queue, and inquiring first service data matched with the first index from a distributed database; and marking a first index in the distributed message queue through the distributed cluster in response to the first service data being queried. Therefore, only the unmarked indexes are obtained from the distributed message queues, the business data are inquired from the distributed database according to the obtained unmarked indexes, and the indexes corresponding to the inquired business data are marked after the business data are inquired successfully, so that the situation that the business data are inquired and obtained repeatedly due to the fact that the indexes corresponding to the inquired business data (namely, the indexes with the marks) are obtained from the distributed message queues when equipment is down or an instance is migrated can be avoided, the data inquiry efficiency can be improved, and the resource occupation can be reduced.

In the technical scheme of the present disclosure, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the related user are all performed under the premise of obtaining the consent of the user, and all meet the regulations of the related laws and regulations, and do not violate the good custom of the public order.

For clarity of explanation of the above embodiments, the present disclosure also proposes a data query method.

Fig. 3 is a schematic flow chart of a data query method according to a second embodiment of the disclosure.

As shown in fig. 3, the data query method may include the steps of:

step 301, obtaining a query condition, and querying a plurality of indexes matched with the query condition from a distributed index database; the distributed index database is used for storing indexes corresponding to all service data in the distributed database.

Step 302, write a plurality of indices to a distributed message queue in a distributed cluster.

For the explanation of steps 301 to 302, reference may be made to the related description in any embodiment of the present disclosure, and details are not repeated herein.

Step 303, obtaining an unmarked first index from the distributed message queue through a plurality of first threads in at least one first instance.

In the embodiment of the present disclosure, the number of the first instances (or referred to as first consumption instances) may be one, or may also be multiple, and the present disclosure does not limit this. Each first instance may include at least one thread (otherwise known as a consuming thread), referred to in this disclosure as a first thread.

In the embodiment of the disclosure, in order to improve the query efficiency of the data, the unmarked first index may be obtained from the distributed message queue through the plurality of first threads in the at least one first instance.

And 304, querying first service data matched with the first index acquired by the corresponding first thread from the distributed database through each first thread.

In the embodiment of the present disclosure, first service data matched with a first index acquired by a corresponding first thread may be queried from a distributed database through each first thread. That is, for any one first thread, first service data matched with the first index acquired by the first thread in the distributed database may be queried according to the first index acquired by the first thread.

Step 305, in response to querying the first service data, marking a first index in the distributed message queue by the distributed cluster.

For the explanation of step 305, reference may be made to relevant descriptions in any embodiment of the present disclosure, which are not described herein again.

According to the data query method, through the plurality of first threads in at least one first instance, unmarked first indexes are obtained from the distributed message queue, and through each first thread, first service data matched with the first indexes obtained by the corresponding first thread are queried from the distributed database. Therefore, the unmarked indexes are obtained from the distributed index database through the multiple threads in at least one instance, and the business data is inquired from the distributed database according to the obtained indexes, so that the data inquiry efficiency can be further improved.

In order to clearly illustrate how the first index in the distributed message queue is marked by the distributed cluster in any embodiment of the present disclosure, the present disclosure also provides a data query method.

Fig. 4 is a schematic flow chart of a data query method provided in the third embodiment of the present disclosure.

As shown in fig. 4, the data query method may include the steps of:

step 401, obtaining a query condition, and querying a plurality of indexes matched with the query condition from a distributed index database; the distributed index database is used for storing indexes corresponding to all service data in the distributed database.

Step 402, writing a plurality of indexes into a distributed message queue in a distributed cluster.

In step 403, a first index without a mark is obtained from the distributed message queue through a plurality of first threads in at least one first instance.

And step 404, inquiring first service data matched with the first index acquired by the corresponding first thread from the distributed database through each first thread.

For the explanation of steps 401 to 404, reference may be made to the related description in any embodiment of the present disclosure, which is not described herein again.

Step 405, in response to the first business data queried by each first thread, combining the first business data queried by each first thread.

In this embodiment of the present disclosure, after each first thread queries or acquires first service data, the first service data queried or acquired by each first thread may be combined (or assembled) to obtain combined first service data.

Step 406, sending the combined first service data to the specified client.

The designated client may be a data receiving end or a data requiring end.

In the embodiment of the present disclosure, the combined first service data may be sent to a specified client.

Step 407, in response to receiving the first acknowledgement response sent by the designated client, marking, by the distributed cluster, the first index acquired by each first thread in the distributed message queue.

The first acknowledgement response is used for indicating that the appointed client receives the combined first service data.

In this embodiment of the present disclosure, when a first acknowledgement response sent by a designated client is received, it may be explicitly specified that the client has successfully received the combined first service data, and at this time, the first index acquired by each first thread in the distributed message queue may be marked by the distributed cluster. For example, a confirmation response may be sent to a service node where a distributed message queue in the distributed cluster is located, and accordingly, after receiving the confirmation response, the service node may mark the first index acquired by each first thread in the distributed message queue.

According to the data query method, the first business data queried by each first thread is combined by responding to the first business data queried by each first thread; sending the combined first service data to a specified client; in response to receiving a first confirmation response sent by a designated client, marking a first index acquired by each first thread in a distributed message queue through a distributed cluster; the first acknowledgement response is used for indicating that the appointed client receives the combined first service data. Therefore, the index corresponding to the successfully received service data in the distributed message queue is marked only when the data receiving end (namely, the designated client) is clear to successfully receive the service data, so that the situation that the service data is lost if the index corresponding to the service data is marked when the data receiving end does not successfully receive the service data and the instance is down or migrated can be avoided.

Fig. 5 is a schematic flow chart of a data query method according to a fourth embodiment of the present disclosure.

As shown in fig. 5, the data query method may include the steps of:

step 501, obtaining query conditions, and querying a plurality of indexes matched with the query conditions from a distributed index library; the distributed index database is used for storing indexes corresponding to all service data in the distributed database.

In any embodiment of the present disclosure, the distributed database is used for storing each service data and an index corresponding to each service data. And the distributed index database is further used for periodically acquiring a third index corresponding to the third service data from the distributed database according to a set period and storing the third index. And the third service data is updated service data and/or newly added service data in the distributed database.

Therefore, the index of each service data in the distributed database can be timely and effectively updated to the distributed index database, so that the comprehensiveness of data query is ensured, and missing query of data is avoided.

Step 502, writing a plurality of indexes into a distributed message queue in a distributed cluster.

Step 503, obtaining the unmarked first index from the distributed message queue through a plurality of first threads in at least one first instance.

Step 504, querying, by each first thread, first service data matched with the first index acquired by the corresponding first thread from the distributed database.

For the explanation of steps 501 to 504, reference may be made to the related description in any embodiment of the present disclosure, which is not described herein again.

Step 505, in response to the first business data queried by any first thread, sending the first business data queried by any first thread to the specified client.

The designated client may be a data receiving end or a data requiring end.

In this embodiment of the present disclosure, when any one of the first threads queries or acquires the first service data, the first thread queries or acquires the first service data may be sent to a specified client.

Step 506, in response to receiving the second acknowledgement response sent by the designated client, marking, by the distributed cluster, the first index obtained by any first thread in the distributed message queue.

And the second confirmation response is used for indicating that the specified client receives the first service data inquired by any first thread.

In this embodiment of the present disclosure, in a case of receiving a second acknowledgement response sent by a specified client, it may be determined that the specified client has successfully received any one of the first thread queries or acquired first service data, and at this time, a first index acquired by any one of the first threads in the distributed message queue may be marked by the distributed cluster. For example, a confirmation response may be sent to a service node where a distributed message queue in the distributed cluster is located, and accordingly, after receiving the confirmation response, the service node may mark a first index acquired by any one of the first threads in the distributed message queue.

In any embodiment of the present disclosure, when each index in the distributed message queue is marked, it indicates that the service data to be queried has been successfully acquired, and in order to reduce resource occupation, each index in the distributed message queue may be deleted. Therefore, the occupied storage resource can be released after the required service data is successfully acquired.

The data query method of the embodiment of the disclosure sends the first business data queried by any first thread to the specified client by responding to the first business data queried by any first thread; in response to receiving a second confirmation response sent by a designated client, marking a first index acquired by any one first thread in the distributed message queue through the distributed cluster; and the second confirmation response is used for indicating the appointed client to receive the first service data inquired by any one first thread. Therefore, after each thread acquires the service data, the service data acquired by each thread can be timely sent to a data demand end (namely, a designated client), and the waiting time of the designated client is reduced.

In a possible implementation manner of the embodiment of the present disclosure, when an instance (denoted as a second instance in the present disclosure) that is down or migrated occurs in each first instance, the second instance may be restarted, and an unmarked index is obtained from the distributed index library by the restarted second instance, so as to achieve the purpose of breakpoint transmission. The above process is described in detail below with reference to fig. 6.

Fig. 6 is a schematic flowchart of a data query method according to a fifth embodiment of the present disclosure.

As shown in fig. 6, on the basis of any one of the embodiments in fig. 2 to fig. 5, the data query method may further include the following steps:

step 601, in response to the second instance being down or migrated in the at least one first instance, restarting the second instance.

In the embodiment of the present disclosure, the number of the second instances (or referred to as second consumption instances) may be one, or may also be multiple, and the present disclosure does not limit this.

In an embodiment of the present disclosure, when a second instance of the at least one first instance is down or migrated, the second instance may be restarted.

Step 602, obtaining a second index without a mark from the distributed message queue through a plurality of second threads in the restarted second instance.

In an embodiment of the present disclosure, the unmarked second index may be obtained from the distributed message queue by a plurality of second threads in the restarted second instance.

Step 603, querying, by each second thread, second service data matched with the second index obtained by the corresponding second thread from the distributed database.

In this embodiment of the present disclosure, second service data that matches the second index obtained by the corresponding second thread may be queried from the distributed database by each second thread. That is, for any second thread, second service data matched with the second index acquired by the second thread in the distributed database may be queried according to the second index acquired by the second thread.

In summary, after the instance is down or migrated, the instance is restarted, so that the query process of the data can be accelerated through the restarted instance, and the QPS of data issuing can be improved.

In a possible implementation manner of the embodiment of the present disclosure, after each second thread queries or acquires second service data, the second service data queried or acquired by each second thread may be combined (or assembled) to obtain combined second service data, and the combined second service data is sent to the specified client.

In a case that the third acknowledgement response sent by the designated client is received, it may be explicitly designated that the client has successfully received the combined second service data, and at this time, the second indexes obtained by the second threads in the distributed message queue may be marked by the distributed cluster. For example, a confirmation response may be sent to the service node where the distributed message queue in the distributed cluster is located, and accordingly, after receiving the confirmation response, the service node may mark the second index obtained by each second thread in the distributed message queue.

Therefore, only when it is clear that the data receiving end (namely, the designated client) has successfully received the service data, the index corresponding to the service data which has been successfully received in the distributed message queue is marked, so that the situation that the index corresponding to the service data is marked when the data receiving end does not successfully receive the service data, and the service data is lost if the instance is down or migrated, can be avoided.

In another possible implementation manner of the embodiment of the present disclosure, when any one second thread queries or acquires second service data, the second thread may send the second service data queried or acquired by the second thread to the specified client.

In a case where a fourth acknowledgement response sent by the designated client is received, it may be determined that the designated client has successfully received the second service data queried by or acquired by any of the second threads, and at this time, the second index acquired by any of the second threads in the distributed message queue may be marked by the distributed cluster. For example, an acknowledgement response may be sent to a service node where a distributed message queue in the distributed cluster is located, and accordingly, after receiving the acknowledgement response, the service node may mark a second index obtained by any one of the second threads in the distributed message queue.

Therefore, after each thread acquires the service data, the service data acquired by each thread can be timely sent to a data demand end (namely, a designated client), and the waiting time of the designated client is reduced.

According to the data query method, the second example is restarted by responding to the crash or migration of the second example in at least one first example; acquiring unmarked second indexes from the distributed message queue through a plurality of second threads in the restarted second instance; and querying second service data matched with the second index acquired by the corresponding second thread from the distributed database through each second thread. Therefore, the embodiment can be restarted after the embodiment is down or migrated, so that the data query process can be accelerated through the restarted embodiment, and the QPS of data issuing can be improved.

As an application scenario, a distributed database is taken as a KV repository, an index of service data is rowkey, and a distributed index repository is an ES (distributed open source search and analysis engine) repository for example, and an implementation principle of any embodiment of the present disclosure may be as shown in fig. 7.

When data is newly added or updated in the KV library, the rowkey in the KV library can be extracted and written into the ES library, and the index of the full service data in the KV library is established through the ES. Wherein, the KV library and the ES library can be regularly synchronized in full quantity.

Taking a distributed message queue as bp (big, a distributed data transmission pipeline, a distributed message queue) for example, when service data in the KV library needs to be synchronized to other service departments, all rowkeys meeting query conditions in the ES library can be queried through the examples, and the rowkeys queried from the ES library are written into bp.

And starting the plurality of consumption examples to consume rowkeys in the bp, wherein the consumption examples can acquire unmarked rowkeys from the bp, then query the KV library in batch according to the acquired unmarked rowkeys to obtain required service data, and assemble the queried service data and then send the assembled service data to clients (marked as designated clients in the disclosure) of other service departments. After the data is successfully sent, an ACK may be fed back to the service node where the bp is located, where the ACK is used to indicate that the rowkey acquired by the consumption instance has been consumed, so that the consumed rowkey may be marked.

In the rowkey consumption process, if the consumption example is down or migrated and no ACK is fed back to the service node where the bp is located, the rowkey in the bp can be consumed at the original position after the consumption example is restarted, namely the rowkey without the mark in the bp is consumed, so that the purpose of breakpoint continuous transmission is achieved, the problem of repeated data transmission is solved, the data query and transmission efficiency can be improved, and the resource occupation can be reduced.

In addition, in the disclosure, the number of the consumption instances and the number of the consumption threads in the consumption instances are configurable, so that the purpose of controlling the QPS according to actual requirements can be achieved.

Corresponding to the data query method provided in the embodiments of fig. 2 to 6, the present disclosure also provides a data query device, and since the data query device provided in the embodiments of the present disclosure corresponds to the data query method provided in the embodiments of fig. 2 to 6, the implementation manner of the data query method is also applicable to the data query device provided in the embodiments of the present disclosure, and will not be described in detail in the embodiments of the present disclosure.

Fig. 8 is a schematic structural diagram of a data query device according to a sixth embodiment of the present disclosure.

As shown in fig. 8, the data query apparatus 800 may include: a query module 801, a write module 802, an obtain module 803, and a mark module 804.

The query module 801 is configured to obtain a query condition, and query a plurality of indexes matching the query condition from the distributed index library; the distributed index database is used for storing indexes corresponding to all service data in the distributed database.

A write module 802 to write a plurality of indices to a distributed message queue in a distributed cluster.

The obtaining module 803 is configured to obtain the unmarked first index from the distributed message queue, and query the first service data matched with the first index from the distributed database.

And the marking module 804 is configured to mark, by the distributed cluster, the first index in the distributed message queue in response to querying the first service data.

In a possible implementation manner of the embodiment of the present disclosure, the obtaining module 803 is configured to: obtaining a first index without marks from a distributed message queue through a plurality of first threads in at least one first instance; and acquiring first service data matched with the first index acquired by the corresponding first thread from the distributed database through each first thread.

In a possible implementation manner of the embodiment of the present disclosure, the marking module 804 is configured to: responding to the first business data inquired by each first thread, and combining the first business data inquired by each first thread; sending the combined first service data to a specified client; in response to receiving a first confirmation response sent by a designated client, marking a first index acquired by each first thread in a distributed message queue through a distributed cluster; the first acknowledgement response is used for indicating that the appointed client receives the combined first service data.

In a possible implementation manner of the embodiment of the present disclosure, the marking module 804 is configured to: responding to the first business data inquired by any first thread, and sending the first business data inquired by any first thread to a specified client; in response to receiving a second confirmation response sent by the appointed client, marking a first index acquired by any first thread in the distributed message queue through the distributed cluster; and the second confirmation response is used for indicating that the specified client receives the first service data inquired by any first thread.

In a possible implementation manner of the embodiment of the present disclosure, the data query apparatus 800 may further include:

and the restarting module is used for responding to downtime or migration of a second instance in the at least one first instance and restarting the second instance.

The obtaining module 803 is further configured to obtain, through the plurality of second threads in the restarted second instance, a second index that is not marked from the distributed message queue;

the obtaining module 803 is further configured to obtain, through each second thread, second service data that is matched with the second index obtained by the corresponding second thread from the distributed database.

and the combination module is used for responding to the second service data inquired by each second thread and combining the second service data inquired by each second thread.

And the first sending module is used for sending the combined second service data to the appointed client.

The marking module 804 is further configured to mark, by the distributed cluster, the second index obtained by each second thread in the distributed message queue in response to receiving the third acknowledgement response sent by the designated client.

And the third confirmation response is used for indicating the appointed client to receive the combined second service data.

and the second sending module is used for responding to the second service data inquired by any second thread and sending the second service data inquired by any second thread to the specified client.

The marking module 804 is further configured to mark, by the distributed cluster, a second index obtained by any second thread in the distributed message queue in response to receiving a fourth acknowledgement response sent by the designated client.

And the fourth confirmation response is used for indicating that the specified client receives the second service data queried by any second thread.

In a possible implementation manner of the embodiment of the present disclosure, the distributed database is used for storing each service data and an index of each service data; the distributed index database is also used for periodically acquiring a third index corresponding to third service data from the distributed database according to a set period and storing the third index; and the third service data is updated service data and/or newly added service data in the distributed database.

and the deleting module is used for deleting each index in the distributed message queue under the condition that each index in the distributed message queue is marked.

The data query device of the embodiment of the disclosure queries a plurality of indexes matched with query conditions from a distributed index database, and writes the plurality of indexes into a distributed message queue in a distributed cluster; acquiring a first index without a mark from a distributed message queue, and inquiring first service data matched with the first index from a distributed database; and marking a first index in the distributed message queue through the distributed cluster in response to the first service data being queried. Therefore, only the unmarked indexes are obtained from the distributed message queues, the business data are inquired from the distributed database according to the obtained unmarked indexes, and the indexes corresponding to the inquired business data are marked after the business data are inquired successfully, so that the situation that the business data are inquired and obtained repeatedly due to the fact that the indexes corresponding to the inquired business data (namely, the indexes with the marks) are obtained from the distributed message queues when equipment is down or an instance is migrated can be avoided, the data inquiry efficiency can be improved, and the resource occupation can be reduced.

To implement the above embodiments, the present disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data query method according to any one of the above embodiments of the disclosure.

In order to achieve the above embodiments, the present disclosure also provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the data query method proposed in any one of the above embodiments of the present disclosure.

In order to implement the foregoing embodiments, the present disclosure further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the data query method proposed in any of the foregoing embodiments of the present disclosure.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 9 illustrates a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 9, the electronic apparatus 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 902 or a computer program loaded from a storage unit 908 into a RAM (Random Access Memory) 903. In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 can be stored. The calculation unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An I/O (Input/Output) interface 905 is also connected to the bus 904.

A number of components in the electronic device 900 are connected to the I/O interface 905, including: an input unit 906 such as a keyboard, a mouse, and the like; an output unit 907 such as various types of displays, speakers, and the like; a storage unit 908 such as a magnetic disk, optical disk, or the like; and a communication unit 909 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the electronic device 900 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 901 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing Unit 901 include, but are not limited to, a CPU (Central Processing Unit), a GPU (graphics Processing Unit), various dedicated AI (Artificial Intelligence) computing chips, various computing Units running machine learning model algorithms, a DSP (Digital Signal Processor), and any suitable Processor, controller, microcontroller, and the like. The computing unit 901 performs the respective methods and processes described above, such as the data query method described above. For example, in some embodiments, the data query methods described above may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 900 via the ROM 902 and/or the communication unit 909. When loaded into RAM 903 and executed by computing unit 901, may perform one or more of the steps of the data query method described above. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the above-described data query method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, FPGAs (Field Programmable Gate arrays), ASICs (Application-Specific Integrated circuits), ASSPs (Application Specific Standard products), SOCs (System On Chip), CPLDs (Complex Programmable Logic devices), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or flash Memory, an optical fiber, a CD-ROM (Compact Disc Read-Only-Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a Display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), internet, and blockchain Network.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in a conventional physical host and a VPS (Virtual Private Server). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be noted that artificial intelligence is a subject for studying a computer to simulate some human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, machine learning/deep learning, a big data processing technology, a knowledge map technology and the like.

Cloud computing (cloud computing) refers to a technology architecture that accesses a flexibly extensible shared physical or virtual resource pool through a network, where resources may include servers, operating systems, networks, software, applications, storage devices, and the like, and may be deployed and managed in an on-demand, self-service manner. Through the cloud computing technology, high-efficiency and strong data processing capacity can be provided for technical application such as artificial intelligence and block chains and model training.

According to the technical scheme of the embodiment of the disclosure, a plurality of indexes matched with query conditions are queried from a distributed index database, and the indexes are written into a distributed message queue in a distributed cluster; acquiring a first index without a mark from a distributed message queue, and inquiring first service data matched with the first index from a distributed database; and marking a first index in the distributed message queue through the distributed cluster in response to the first service data being queried. Therefore, only the unmarked indexes are obtained from the distributed message queues, the service data are inquired from the distributed database according to the obtained unmarked indexes, and the indexes corresponding to the inquired service data are marked after the service data are inquired successfully, so that the situation that the service data are repeatedly inquired and obtained due to the fact that the indexes corresponding to the inquired service data (namely the indexes with the marks) are obtained from the distributed message queues when equipment is down or an instance is migrated can be avoided, the data inquiry efficiency can be improved, and the resource occupation can be reduced.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions proposed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method of data query, comprising:

2. The method of claim 1, wherein the retrieving an unmarked first index from the distributed message queue and querying the distributed database for first traffic data matching the first index comprises:

obtaining the first index from the distributed message queue without a tag by a plurality of first threads in at least one first instance;

and querying first service data matched with the first index acquired by the corresponding first thread from the distributed database through each first thread.

3. The method of claim 2, wherein said marking the first index in the distributed message queue by the distributed cluster in response to querying the first traffic data comprises:

responding to the first business data inquired by each first thread, and combining the first business data inquired by each first thread;

sending the combined first service data to a specified client;

in response to receiving a first confirmation response sent by the designated client, marking a first index acquired by each first thread in the distributed message queue through the distributed cluster;

wherein the first acknowledgement response is used to indicate that the specified client receives the combined first service data.

4. The method of claim 2, wherein said marking the first index in the distributed message queue by the distributed cluster in response to querying the first traffic data comprises:

responding to the first business data inquired by any first thread, and sending the first business data inquired by any first thread to a specified client;

in response to receiving a second confirmation response sent by the designated client, marking a first index acquired by any first thread in the distributed message queue through the distributed cluster;

the second confirmation response is used for indicating the designated client to receive the first service data queried by any first thread.

5. The method of claim 2, wherein the method further comprises:

restarting a second instance of the at least one first instance in response to the second instance being down or migrated;

obtaining a second index without a mark from the distributed message queue through a plurality of second threads in the restarted second instance;

and querying second service data matched with a second index acquired by the corresponding second thread from the distributed database through each second thread.

6. The method of claim 5, wherein the method further comprises:

responding to second service data inquired by each second thread, and combining the second service data inquired by each second thread;

sending the combined second service data to the appointed client;

marking, by the distributed cluster, a second index obtained by each second thread in the distributed message queue in response to receiving a third acknowledgement response sent by the designated client;

wherein the third acknowledgement response is used to indicate that the designated client receives the combined second service data.

7. The method of claim 5, wherein the method further comprises:

responding to any second thread inquired second service data, and sending the second service data inquired by any second thread to a specified client;

in response to receiving a fourth acknowledgement response sent by the designated client, marking, by the distributed cluster, a second index obtained by any one of the second threads in the distributed message queue;

wherein the fourth acknowledgement response is used to indicate that the designated client receives the second service data queried by any of the second threads.

8. The method of any one of claims 1-7, wherein the distributed database is configured to store each of the business data and an index of each of the business data;

the distributed index database is further used for periodically obtaining a third index corresponding to third service data from the distributed database according to a set period, and storing the third index;

and the third service data is updated service data and/or newly added service data in the distributed database.

9. The method according to any one of claims 1-7, wherein the method further comprises:

deleting each index in the distributed message queue if the index in the distributed message queue is marked.

10. A data query device, comprising:

the query module is used for acquiring query conditions and querying a plurality of indexes matched with the query conditions from the distributed index database; the distributed index database is used for storing indexes corresponding to all service data in the distributed database;

and the marking module is used for marking the first index in the distributed message queue through the distributed cluster in response to the first service data being inquired.

11. The apparatus of claim 10, wherein the means for obtaining is configured to:

and acquiring first service data matched with the first index acquired by the corresponding first thread from the distributed database through each first thread.

12. The apparatus of claim 11, wherein the tagging module is to:

sending the combined first service data to a specified client;

marking a first index acquired by each first thread in the distributed message queue through the distributed cluster in response to receiving a first confirmation response sent by the designated client;

13. The apparatus of claim 11, wherein the tagging module is to:

and the second confirmation response is used for indicating the designated client to receive the first service data queried by any first thread.

14. The apparatus of claim 11, wherein the apparatus further comprises:

a restart module, configured to restart a second instance of the at least one first instance in response to the second instance being down or migrated;

the obtaining module is further configured to obtain, through the restarted plurality of second threads in the second instance, a second index that is not marked from the distributed message queue;

the obtaining module is further configured to obtain, through each second thread, second service data matched with a second index obtained by a corresponding second thread from the distributed database.

15. The apparatus of claim 14, wherein the apparatus further comprises:

the combination module is used for responding to second service data inquired by each second thread and combining the second service data inquired by each second thread;

the first sending module is used for sending the combined second service data to the appointed client;

the marking module is further configured to mark, by the distributed cluster, a second index obtained by each second thread in the distributed message queue in response to receiving a third acknowledgement response sent by the designated client;

16. The apparatus of claim 14, wherein the apparatus further comprises:

the second sending module is further used for responding to second service data inquired by any second thread and sending the second service data inquired by any second thread to a specified client;

the marking module is further configured to mark, by the distributed cluster, a second index obtained by any one of the second threads in the distributed message queue in response to receiving a fourth acknowledgement response sent by the designated client;

17. The apparatus according to any one of claims 10 to 16, wherein the distributed database is configured to store each of the business data and an index of each of the business data;

the distributed index database is further used for periodically acquiring a third index corresponding to third service data from the distributed database according to a set period, and storing the third index;

18. The apparatus of any one of claims 10-16, wherein the apparatus further comprises:

a deleting module, configured to delete each index in the distributed message queue when each index in the distributed message queue is marked.

19. An electronic device, wherein the electronic device comprises:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-9.

21. A computer program product comprising a computer program which, when being executed by a processor, carries out the steps of the method according to any one of claims 1-9.