CN113297269A

CN113297269A - Data query method and device

Info

Publication number: CN113297269A
Application number: CN202110183730.XA
Authority: CN
Inventors: 张天雨
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2021-02-10
Filing date: 2021-02-10
Publication date: 2021-08-24

Abstract

The present specification provides a data query method and apparatus, wherein the data query method includes: acquiring a data query request, wherein the data query request carries a partition key and query attribute information; determining a target partition corresponding to the partition key in a first database, and reading attribute information of a data block contained in the target partition; under the condition that the query attribute information is not matched with the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information; and determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

Description

Data query method and device

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data query method and apparatus.

Background

With the development of internet technology, more and more information starts to be digitalized, and in order to meet the backtracking request of a user, a part of online service platforms provide log storage service for the user, so that when the user backtracks related data, the user related data can be read and fed back through the log storage service provided by the online service platforms. In the prior art, a log database supporting multiple tenants and used by an online service platform is stored with related data in a column-stored data file format, and although the storage and retrieval of the related data can be realized, the data is stored in the column-stored data file format, and an index structure and an effective retrieval manner cannot be provided, so that the log database cannot meet the high write-in flow and has the problem of low retrieval performance, and an effective scheme is urgently needed to solve the problem.

Disclosure of Invention

In view of this, the present specification provides a data query method. The present specification also relates to a data query apparatus, a computing device, and a computer-readable storage medium to solve the technical problems in the prior art.

According to a first aspect of embodiments of the present specification, there is provided a data query method including:

acquiring a data query request, wherein the data query request carries a partition key and query attribute information;

determining a target partition corresponding to the partition key in a first database, and reading attribute information of a data block contained in the target partition;

under the condition that the query attribute information is not matched with the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

and determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

Optionally, before the step of obtaining the data query request is executed, the method further includes:

extracting data to be migrated with the storage time larger than a preset time threshold value in the first database, and determining a migration partition key corresponding to the data to be migrated;

compressing the data to be migrated into compressed data, and writing the compressed data into a partition corresponding to the migration partition key in the second database;

determining compression attribute information and compression index information corresponding to the compressed data according to a writing result, and establishing a storage corresponding relation between the migration partition key and the compression attribute information as well as the compression index information;

and updating the storage relation table corresponding to the second database based on the storage corresponding relation.

Optionally, the determining, according to the writing result, compression index information corresponding to the compressed data includes:

judging whether the compressed data meets a preset index creation condition or not;

and if so, creating a compression index corresponding to the compressed data according to the writing result, and determining the compression index information based on the compression index.

Optionally, the creating a compression index corresponding to the compressed data according to the writing result includes:

determining a data type of the compressed data;

under the condition that the data type is a text type, creating an inverted index corresponding to the compressed data according to a writing result, and taking the inverted index as the compressed index;

and under the condition that the data type is a numerical value type, creating a tree index corresponding to the compressed data according to a writing result, and using the tree index as the compressed index.

Optionally, before the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data block included in the target partition is executed, the method further includes:

detecting whether the query attribute information contains time information or not;

if not, executing the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data block contained in the target partition.

Optionally, if the determination result of detecting whether the query attribute information includes the time information is yes, the following steps are performed:

judging whether the time information meets a preset time division condition or not;

if yes, determining a target partition corresponding to the partition key in the first database, and reading attribute information of a data block contained in the target partition;

and matching the attribute information with the query attribute information, and reading target data corresponding to the data query request in the data block according to a matching result.

Optionally, if the determination result of determining whether the time information meets the preset time division condition is negative, the following steps are performed:

reading the storage relation table corresponding to the second database according to the partition key and the query attribute information;

Optionally, before the step of determining, when the query attribute information is not matched with the attribute information, a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information is executed, the method further includes:

analyzing the attribute information to obtain a first numerical interval, and analyzing the query attribute information to obtain a second numerical interval;

judging whether the first numerical value interval comprises the second numerical value interval or not;

if yes, reading target data corresponding to the data query request in the data block based on the query attribute information;

if not, executing the step of determining the second database having the data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information.

Optionally, the reading the storage relationship table corresponding to the second database according to the partition key and the query attribute information includes:

performing hash operation on the partition key and the query attribute information to obtain a query hash value;

determining candidate partition keys and time interval information corresponding to the query hash value in a hash table corresponding to the second database;

determining a second target partition in the second database according to the candidate partition key and the time interval information, and reading second attribute information of a candidate data block contained in the second target partition;

and reading the storage relation table corresponding to the second database according to the query attribute information under the condition that the second attribute information is matched with the query attribute information.

Optionally, the determining the index information according to the reading result includes:

taking the query index as the index information under the condition of obtaining the query index according to the reading result;

correspondingly, the reading, in the second database, target data corresponding to the data query request based on the index information includes:

and reading target data corresponding to the data query request in the second database based on the query index.

According to a second aspect of embodiments of the present specification, there is provided a data query apparatus including:

the acquisition request module is configured to acquire a data query request, wherein the data query request carries a partition key and query attribute information;

the reading information module is configured to determine a target partition corresponding to the partition key in a first database and read attribute information of a data block contained in the target partition;

the determining database module is configured to determine a second database having a data migration relationship with the first database under the condition that the query attribute information is not matched with the attribute information, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

and the data reading module is configured to determine index information according to a reading result, and read target data corresponding to the data query request in the second database based on the index information.

According to a third aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is to store computer-executable instructions, and the processor is to execute the computer-executable instructions to:

According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data query method.

In the data query method provided by the present specification, under the condition of acquiring the data carrying the partition key and the query attribute information, the target partition corresponding to the partition key is determined in the first database, the attribute information of the data block contained in the target partition is read, under the condition that the query attribute information is not matched with the attribute information, the second database having a data migration relationship with the first database can be determined, meanwhile, the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information, the index information can be determined according to the reading, and finally, the target data corresponding to the data query request is read in the second database based on the index information, so that the problem that the write-in flow is large and the data storage cannot be performed is solved by combining the first database and the second database, and meanwhile, the data query is performed in different manners for the first database and the second database, the data query efficiency is effectively improved, and therefore quick response to the data query request is realized.

Drawings

FIG. 1 is a flow chart of a data query method provided in an embodiment of the present specification;

FIG. 2 is a diagram illustrating an cold data format in a data query method according to an embodiment of the present disclosure;

FIG. 3 is a flowchart illustrating a data query method applied in a log audit scenario according to an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of a data query device according to an embodiment of the present disclosure;

fig. 5 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

A tenant: refers to a user using a Log Storage service (Log Storage) provided by a service provider, where the same Log Storage can be used by multiple tenants.

Thermal data: the data structure of the hot data is write-optimized.

Cold data: the data structure of cold data is read-optimized, and the data structure of the cold data is read-optimized.

In the present specification, a data query method is provided, and the present specification relates to a data query apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

In practical application, in order to improve the calculation performance and storage efficiency of Hadoop Hive, data storage is usually performed in an ORC column storage data format, where the format usually includes Postscript, File font, Stripe data blocks, and column statistics, so as to store data, and although File-level retrieval and filtering can be achieved, due to the characteristics of the ORC column storage data format, a data structure for write optimization of real-time data cannot be provided, and in an audit log scenario of log storage service, the ORC itself does not provide an index structure and a retrieval manner that satisfy fast data retrieval, so that the method cannot be applied to the audit log scenario, and therefore an effective scheme is urgently needed to solve the above problems.

Fig. 1 shows a flowchart of a data query method provided in an embodiment of the present specification, which specifically includes the following steps:

step S102, a data query request is obtained, and the data query request carries a partition key and query attribute information.

In practical application, due to the characteristics of online services, most of the online services establish a log record for recording service-related data and corresponding service processing processes, and as the log data continuously increase, a server side is difficult to store a large amount of log data, so that the log storage service side provides log data storage service for tenants in order to ensure that the server side can retain a large amount of log data, and the log data can be traced back at any time by establishing an exclusive database for the tenants in a log data storage manner, so that the audit of the log data is met.

Based on the data query request, the data query request specifically refers to a request for querying related data of the log in an audit log scene, and is submitted to a log storage service party by a tenant; correspondingly, the partition key specifically refers to an ID corresponding to a tenant, and is used for determining a partition corresponding to the tenant in a database, it should be noted that, in order to improve the utilization rate of a log database, the same log database may provide log data storage services for multiple tenants at the same time, each tenant corresponds to one partition, the key of each partition is the ID of the tenant, and the ID of the tenant has uniqueness; correspondingly, the query attribute information specifically refers to attributes of the data itself, including but not limited to a maximum value, a line number, an occupied space, a type, storage time, generation time, and the like of the data.

Further, in the case that the data query request is received, it indicates that the tenant needs to query the historical log data, and before that, in order to improve the query performance, the data in different time periods may be stored in different manners, so as to ensure that the log storage service provider can quickly respond to different query requests of different tenants, in this embodiment, the specific implementation manner is as follows:

Specifically, the preset time threshold is a condition for judging data migration, it can be determined that the data needs to be converted from hot data into cold data if the storage time in the first database exceeds the time threshold, it can be determined that the current state of the data is the hot data if the storage time in the first database does not exceed the time threshold, accordingly, the first database is specifically hot data whose storage write time does not exceed the preset time threshold, the second database is specifically cold data whose storage write time exceeds the preset time threshold, and the cold data is obtained by converting the hot data whose storage write time exceeds the preset time threshold.

It should be noted that, because the first database and the second database are both log databases for storing log data, the first database is a database for storing hot data, the second database is a database for storing cold data, and the cold data is obtained by converting the hot data, the second database and the first database contain partitions corresponding to the same partition key, that is, the same partition key may determine a partition in the first database, or may determine a partition in the second database.

In addition, because the data stored in the first database is hot data, and the conversion of the hot data into cold data consumes more computing resources, after the data is received, the data is written into the first database first, and when the storage time in the first database exceeds a preset time threshold, the hot data in the first database is converted into cold data to be stored in the second database; further, in order to improve the query performance, the format of the data stored in the second database may be composed of three parts, namely, meta information, index and data, as shown in fig. 2, the meta information includes information such as table meta information, list of columns, index list, and the like, the index includes index meta information, index data, and the like, and in order to improve the query performance, an inverted index may be established for text type data, a kd tree index may be established for value type, and the data includes statistical information (maximum value, number of rows) of data, compressed original data, and the like.

The first database and the second database both comprise partitions corresponding to partition keys, the compressed data specifically refers to data obtained by compressing data to be migrated, and the occupied space of the compressed data is smaller than that of the data to be migrated; the compression attribute information specifically refers to the maximum value, the line number and the like of the data to be migrated, and the compression index information specifically refers to information on whether an index is provided.

Based on this, the data storage time in the first database is detected at set time intervals, and when the data needing to be migrated is detected, the data to be migrated, the storage time of which is greater than a preset time threshold, is extracted from the first database, and a migration partition key corresponding to the data to be migrated is determined at the same time, wherein the migration partition key specifically refers to a key corresponding to a partition in which the data to be migrated is stored; then, in order to improve subsequent query performance, the data to be migrated may be compressed to obtain compressed data, and the compressed data is written into the partition corresponding to the migration partition key in the second database; after the writing is finished, in order to facilitate the follow-up data query, the compression attribute information and the compression index information of the compressed data can be determined, then the storage corresponding relation between the migration partition key and the compression attribute information and the compression index information is established, and finally the storage relation table corresponding to the second database is updated based on the storage corresponding relation; the storage relation table of the second database records the corresponding relation among the partition keys, the data attribute information and the index information, and when data is searched, the information needing to be read can be determined through the reading table.

Furthermore, after the compressed data is written into the second database, an index may be created for the compressed data according to the writing result, and in the process of creating the index, since data related to the index also needs to occupy a storage space, in consideration of space utilization, different compressed data may select whether to create the index according to a requirement, in this embodiment, a specific implementation manner is as follows:

if yes, determining the data type of the compressed data; under the condition that the data type is a text type, creating an inverted index corresponding to the compressed data according to a writing result, and taking the inverted index as the compressed index; and under the condition that the data type is a numerical value type, creating a tree index corresponding to the compressed data according to a writing result to serve as the compressed index, and determining the compressed index information based on the compressed index.

And if not, taking the information without the index as the compressed index information.

Specifically, the index creating condition is a condition for judging whether to create an index for index data, where the index creating condition may be to judge whether an occupied space of compressed data is greater than a preset occupied threshold, or to judge whether a length of the compressed data is greater than a preset length threshold; correspondingly, because the structures of different types of data are different, the index is created in different ways for different types of data when the index creation condition is satisfied.

Based on this, since the data corresponding to the index also needs to occupy a certain space, when determining whether to create the index, it can be determined whether the created index is larger than the occupied space of the compressed data, if so, the creation is not needed, and if not, the creation is needed; further, after the compressed data is written into the second database, whether the compressed data meets a preset index creation condition is judged, if yes, the data type of the compressed data can be determined, under the condition that the data type is a text type, an inverted index corresponding to the compressed data can be created according to a writing result, and the inverted index is used as the compressed index; under the condition that the data type is a numerical value type, creating a tree index corresponding to the compressed data according to a writing result, and using the tree index as the compressed index; finally, the compression index information can be determined based on the compression index.

For example, the log data generated by tenant 1 at a certain moment is stored in the hot database for more than 1 hour, and it is determined that the log data needs to be converted into cold data to be stored in the cold database; based on the above, determining that the migration partition key corresponding to the log data to be migrated is cid _1, determining the partition corresponding to the migration partition key cid _1 in the cold data, at this time, compressing the log data to be migrated by adopting a preset compression algorithm to obtain compressed log data, and then writing the compressed log data into the migration partition keyA partition corresponding to cid _ 1; after the writing is finished, in order to audit the log subsequently, the compression attribute information of the compressed log data is determined to be { maximum value L_maxMinimum value L_minThe method comprises the steps of line number N, storage time T and creation time TS }, determining a kd tree index established by compressed log data, determining that the compressed index information is provided with an index, then establishing a storage corresponding relation between a migration partition key cid _1 and compressed attribute information and compressed index information, and writing the storage corresponding relation into a storage relation table corresponding to a cold database, so that corresponding log data can be quickly inquired in the cold database in the subsequent log audit process.

Wherein the inverted index is used to store a mapping of the storage location of a word in a document or a group of documents under a full-text search. Which is the most common data structure in document retrieval systems. By inverted indexing, a list of documents containing a word can be quickly retrieved from that word.

In conclusion, by means of periodically converting the write-in optimized data structure (hot data) into the read-out optimized data structure (cold data), the efficiency of querying the data in the second database is effectively improved, so that query requests of different tenants can be quickly responded, and the use experience of the tenants is improved.

Step S104, determining a target partition corresponding to the partition key in a first database, and reading attribute information of a data block contained in the target partition.

Specifically, on the basis of the obtained data query request, a target partition is determined in the first database according to a partition key carried in the data query request, where the target partition specifically refers to a partition storing log data of a tenant, and after the target partition is determined, in order to perform feedback for the data query request, data information of a data block included in the target partition may be read at this time, where the data block specifically refers to a data block composed of partial log data, and the attribute information of the data block specifically refers to a maximum value, a line number, an occupied space, a type, storage time, generation time, and the like corresponding to data stored in the data block, so as to be used for subsequent query processing operations.

Further, because different data query requests provide different query conditions, if the received data query request includes more information, the database that needs to be queried may be accurately determined and stored according to the data query request, and the query may be performed, if the received data query request includes less information, the database may not be accurately determined according to the information included in the data query request, and the query may be performed in the manner from step S102 to step S108, in this embodiment, how to select a correct query manner according to the data query request, and a specific implementation manner is as follows:

If so, judging whether the time information meets the preset time division condition again;

if yes, determining a target partition corresponding to the partition key in the first database, and reading attribute information of a data block contained in the target partition; and matching the attribute information with the query attribute information, and reading target data corresponding to the data query request in the data block according to a matching result.

If not, reading the storage relation table corresponding to the second database according to the partition key and the query attribute information; and determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

Specifically, the time information specifically refers to time-related information contained in the query attribute information, and a time node corresponding to data to be queried can be determined through the time information; correspondingly, the preset time division condition specifically refers to a condition for judging that the data to be queried is stored in the first database or the second database.

Based on this, after the data query request is obtained, it may be detected whether the query attribute information includes time information, and if not, it indicates that the database to be queried cannot be accurately determined according to the data query request, step S104 may be executed; if yes, the database corresponding to the data to be queried can be preliminarily determined and stored through the time information contained in the query attribute information.

When it is determined that the query attribute information includes the time information, it may be determined whether the time information included in the query attribute information meets a preset time division condition, if yes, it is indicated that target data corresponding to the data query request may be queried in the first database, a target partition may be determined in the first database according to a partition key included in the data query request, then attribute information of a data block included in the target partition is read, the attribute information and the query attribute information are simultaneously matched, a target data block satisfying the data query request may be determined to be stored according to a matching result, and finally, the target data is read from the target data block and fed back according to the data query request.

If the data query request does not conform to the first database, it is determined that the target data corresponding to the data query request can be queried in the second database, and because the second database is a database storing cold data, the storage relationship table corresponding to the second database can be read according to the partition key and the query attribute information at this time, the index information corresponding to the data query request can be determined according to the read result, and finally, the target data corresponding to the data query request can be read in the second database based on the index information, and the target data is fed back as a response corresponding to the data query request.

Along the above example, when the log data of tenant 1 is required to be audited, a data query request is submitted at this time, the query attribute information contained in the data query request is analyzed, it is determined that the time information contained in the query attribute information is T1, and at this time, it is determined that the log data required to be audited is stored in the database by analyzing the time information; based on this, at this time, it is determined whether the time information T1 included in the query attribute information meets a preset time division condition;

if the log data which needs to be audited is stored in the hot database, at this time, the target partition may be determined in the hot database according to the partition key cid _1 included in the data query request, then the attribute information of the data block included in the target partition is read, the attribute information is determined to be (maximum value 10, minimum value 0, line number N1, time T2), the query attribute information is (maximum value 5, minimum value 0, line number N2, time T1), the data corresponding to the data query request exists in the data block is determined by comparing the query attribute information in the data query request with the data information of the data block, and at this time, the target log data which meets the query attribute information may be extracted from the data block according to the query attribute information, and feedback is performed for performing subsequent auditing processing according to the data query request.

If the log data are not matched, the log data needing to be audited are stored in the cold database, at this time, a storage relation table corresponding to the cold database can be read according to the partition key cid _1 and the query attribute information (the maximum value is 5, the minimum value is 0, the line number is N2, and the time is T1), the index of the target log data is determined according to the reading result, then the target log data meeting the data query request are obtained in a loading and index retrieval mode, and feedback is carried out according to the data query request so as to be used for carrying out subsequent auditing processing.

In summary, before data query is performed, in order to improve query efficiency, a query mode required to be adopted can be accurately distinguished through time information, so that target data can be quickly fed back according to the data query request, query performance under a log audit scene is improved, and log audit efficiency is improved.

Step S106, under the condition that the query attribute information is not matched with the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information.

Specifically, on the basis of determining the attribute information of the data block, at this time, the query attribute information may be matched with the attribute information, if the query attribute information is matched with the attribute information, it indicates that target data corresponding to the data query request exists in the first database, and target data corresponding to the data query request may be queried in the first database, that is, by filtering PartitionId maps of records cid (partition key) and row _ count (row number), a target partition corresponding to the partition key is obtained, and then for columns (columns) other than the partition key in the data query request, preliminary filtering may be performed according to hot block statistics (including min and max values), if complete filtering is not possible, it indicates that target data meeting the data query request may be queried in the data block, deep filtering can be performed by scanning the data blocks, so as to obtain target data corresponding to the data query request. If the query attribute information and the attribute information are not matched, it is indicated that data corresponding to the data query request does not exist in the first database, at this time, a second database having a data migration relationship with the first database may be determined, and query processing of target data is continued.

It should be noted that, the data migration relationship specifically means that the data in the first database is converted into the second database when a condition is met, and a specific implementation manner may refer to corresponding description in the foregoing embodiment, which is not limited herein.

Based on this, after the second database is determined, since the second database is a database for storing cold data, during query, the storage relationship table corresponding to the second database needs to be read to determine to store the compressed data of the data block corresponding to the data query request and the corresponding index information, so as to directly read the compressed data through the index and decompress the compressed data to obtain the target data corresponding to the data query request.

Further, in the process of determining whether the query attribute information and the attribute information are matched, in order to improve matching accuracy and achieve quick feedback of target data for the data query request, the determination may be performed according to a value of the data, and in this embodiment, a specific implementation manner is as follows:

if not, go to step S106.

Specifically, the first numerical interval specifically refers to an interval formed by a maximum value and a minimum value of data values in a data block, and the second numerical interval specifically refers to a maximum value and a minimum value corresponding to data to be queried; on the basis, a first numerical value interval is obtained by analyzing the attribute information, and a second numerical value interval is obtained by analyzing the query attribute information; then, whether the second numerical interval is included in the first numerical interval is judged, if not, it is determined that the attribute information and the query attribute information are not matched, and at this time, step S106 may be continuously executed to determine target data corresponding to the data query request in a second database; if so, determining that the attribute information is matched with the query attribute information, and reading target data corresponding to the data query request in the data block based on the query attribute information, namely obtaining data which is the same as the query attribute information in the first database as the target data.

In summary, in the process of querying the target data corresponding to the data query request, a manner of comparing the first numerical value interval with the second numerical value interval is adopted, so that whether the target data corresponding to the data query request exists in the first database can be accurately determined, and a manner of traversing all data contained in the database is saved, so as to save query time.

Furthermore, in the process of reading the storage relationship table corresponding to the second database according to the partition key and the query attribute information, because the second database belongs to the cold database, the second database has higher query efficiency than the first database, and in order to further improve the efficiency of querying data in the second database and reduce the occupied space of other data except the data itself, the corresponding information may be recorded in a hash table manner, in this embodiment, the specific implementation manner is as follows:

Specifically, under the condition that it is determined that the target data exists in the second database, hash operation may be performed on the partition key and the query attribute information to obtain a query hash value, and then candidate partition keys and time interval information corresponding to the query hash value are determined in a hash table corresponding to the second database; then, a second target partition for storing target data is determined in the second database according to the candidate partition key and the time interval information, and since the second target partition contains a large number of data blocks, a candidate data block can be determined in the second target partition according to the query attribute information and the time interval information, at this time, the determined data is stored in the candidate data block, then, second attribute information of the candidate data block is read, the second attribute information is matched with the query attribute information, and when the second attribute information is matched with the query attribute information, it indicates that target data corresponding to the data query request exists in the second database, then, a storage relation table corresponding to the second database can be further read according to the query attribute information to determine related information of the stored target data, for subsequent query processing operations.

In practical application, in the process of querying data in the second database, one or more data blocks to be queried may be obtained by filtering according to cid and < min _ ts, max _ ts > information stored in a log block map during querying, and then, since each data block may store statistics (maximum, minimum, null set, etc.) of each column, for columns except for partition keys in a query condition, further filtering may be performed according to query attribute information in a data query request, so as to determine whether target data corresponding to the data query request can be queried in the second database, and if there is description attribute information matching success, subsequent query processing may be performed.

In summary, the relevant information in the second database is recorded by adopting a hash table, so that not only can the occupied space of the relevant information be reduced, but also the information can be prevented from being tampered, and the efficiency of data query in the second database is improved.

Step S108, determining index information according to the reading result, and reading the target data corresponding to the data query request in the second database based on the index information.

Specifically, after the storage relationship table corresponding to the second database is queried according to the partition key and the query attribute information, the index information may be determined according to a reading result, where the index information may be information including an index or not including an index, and after the index information is determined, the target data corresponding to the data query request may be read in the second database according to the index information.

Further, in the process of querying the target data according to the index information, because there are two cases in the index information, different manners will be adopted for querying the target data, and in this embodiment, the specific implementation manner is as follows:

taking the query index as the index information under the condition of obtaining the query index according to the reading result; and reading target data corresponding to the data query request in the second database based on the query index.

Under the condition that the query index is not obtained according to the query result, the data block matched with the query attribute information can be selected from the second database, and then the target data corresponding to the data query request is selected from the data block for feedback.

The following description further explains the data query method provided in this specification by taking an application of the data query method in a log audit scenario as an example, with reference to fig. 3. Fig. 3 shows a processing flow chart of a data query method applied to a log audit scenario provided in an embodiment of the present specification, and specifically includes the following steps:

step S302, extracting the data to be migrated with the storage time larger than the preset time threshold value in the first database, and determining a migration partition key corresponding to the data to be migrated.

Step S304, compressing the data to be migrated into compressed data, and writing the compressed data into the partition corresponding to the migration partition key in the second database.

Step S306, determining the compression attribute information and the compression index information corresponding to the compression data according to the writing result, and establishing the storage corresponding relation between the migration partition key and the compression attribute information and the compression index information.

And step S308, updating the storage relation table corresponding to the second database based on the storage corresponding relation.

Step S310, a data query request is obtained, and the data query request carries a partition key and query attribute information.

Step S312, determining a target partition corresponding to the partition key in the first database, and reading attribute information of the data block included in the target partition.

And step S314, determining a second database having a data migration relationship with the first database under the condition that the query attribute information is not matched with the attribute information.

Step S316, perform hash operation on the partition key and the query attribute information to obtain a query hash value.

Step S318, determining candidate partition keys and time interval information corresponding to the query hash value in the hash table corresponding to the second database.

Step S320, determining a second target partition in the second database according to the candidate partition key and the time interval information, and reading second attribute information of the candidate data block included in the second target partition.

Step S322, reading the storage relationship table corresponding to the second database according to the query attribute information when the second attribute information matches the query attribute information.

Step S324, determining the index information according to the reading result, and reading the target data corresponding to the data query request in the second database based on the index information.

Corresponding to the above method embodiment, this specification further provides a data query apparatus embodiment, and fig. 4 shows a schematic structural diagram of a data query apparatus provided in an embodiment of this specification. As shown in fig. 4, the apparatus includes:

an obtaining request module 402, configured to obtain a data query request, where the data query request carries a partition key and query attribute information;

a read information module 404 configured to determine a target partition corresponding to the partition key in a first database, and read attribute information of a data block included in the target partition;

a determining database module 406, configured to determine a second database having a data migration relationship with the first database if the query attribute information is not matched with the attribute information, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

and a data reading module 408 configured to determine index information according to the reading result, and read target data corresponding to the data query request in the second database based on the index information.

In an optional embodiment, the data query apparatus further includes:

the migration module is configured to extract data to be migrated from the first database, wherein the storage time of the data to be migrated is greater than a preset time threshold, and determine a migration partition key corresponding to the data to be migrated; compressing the data to be migrated into compressed data, and writing the compressed data into a partition corresponding to the migration partition key in the second database; determining compression attribute information and compression index information corresponding to the compressed data according to a writing result, and establishing a storage corresponding relation between the migration partition key and the compression attribute information as well as the compression index information; and updating the storage relation table corresponding to the second database based on the storage corresponding relation.

In an optional embodiment, the migration module is further configured to:

judging whether the compressed data meets a preset index creation condition or not; and if so, creating a compression index corresponding to the compressed data according to the writing result, and determining the compression index information based on the compression index.

In an optional embodiment, the migration module is further configured to:

determining a data type of the compressed data; under the condition that the data type is a text type, creating an inverted index corresponding to the compressed data according to a writing result, and taking the inverted index as the compressed index; and under the condition that the data type is a numerical value type, creating a tree index corresponding to the compressed data according to a writing result, and using the tree index as the compressed index.

In an optional embodiment, the data query apparatus further includes:

a detection module configured to detect whether the query attribute information includes time information; if not, the read information module 404 is executed.

In an optional embodiment, if the determination result of the detection is yes, the following modules are operated:

the first judgment module is configured to judge whether the time information meets a preset time division condition; if yes, determining a target partition corresponding to the partition key in the first database, and reading attribute information of a data block contained in the target partition; and matching the attribute information with the query attribute information, and reading target data corresponding to the data query request in the data block according to a matching result.

In an optional embodiment, the first determining module is further configured to:

reading the storage relation table corresponding to the second database according to the partition key and the query attribute information; and determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

In an optional embodiment, the data query apparatus further includes:

the analysis module is configured to analyze the attribute information to obtain a first numerical interval and analyze the query attribute information to obtain a second numerical interval;

a second judging module configured to judge whether the first value interval includes the second value interval; if yes, reading target data corresponding to the data query request in the data block based on the query attribute information;

if not, the determine database module 406 is run

In an optional embodiment, the determining database module 406 is further configured to:

performing hash operation on the partition key and the query attribute information to obtain a query hash value; determining candidate partition keys and time interval information corresponding to the query hash value in a hash table corresponding to the second database; determining a second target partition in the second database according to the candidate partition key and the time interval information, and reading second attribute information of a candidate data block contained in the second target partition; and reading the storage relation table corresponding to the second database according to the query attribute information under the condition that the second attribute information is matched with the query attribute information.

In an alternative embodiment, the read data module 408 is further configured to:

The data query device provided in this embodiment determines, when the data query device obtains the data block carrying the partition key and the query attribute information, the target partition corresponding to the partition key in the first database, reads the attribute information of the data block included in the target partition, and when the query attribute information is not matched with the attribute information, at this time, the second database having a data migration relationship with the first database may be determined, and simultaneously, the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information, and the index information may be determined according to the reading, and finally, the target data corresponding to the data query request is read in the second database based on the index information, so that the problem that the data storage cannot be performed due to a large write flow is solved by combining the first database and the second database, and meanwhile, data query is performed in different manners for the first database and the second database, the data query efficiency is effectively improved, and therefore quick response to the data query request is realized.

The above is an exemplary scheme of a data query apparatus of the present embodiment. It should be noted that the technical solution of the data query apparatus and the technical solution of the data query method belong to the same concept, and details that are not described in detail in the technical solution of the data query apparatus can be referred to the description of the technical solution of the data query method.

Fig. 5 illustrates a block diagram of a computing device 500 provided according to an embodiment of the present description. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is coupled to memory 510 via bus 530, and database 550 is used to store data.

Computing device 500 also includes access device 540, access device 540 enabling computing device 500 to communicate via one or more networks 560. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 540 may include one or more of any type of network interface, e.g., a Network Interface Card (NIC), wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 500, as well as other components not shown in FIG. 5, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 5 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 500 may also be a mobile or stationary server.

Wherein processor 520 is configured to execute the following computer-executable instructions:

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data query method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data query method.

An embodiment of the present specification also provides a computer readable storage medium storing computer instructions that, when executed by a processor, are operable to:

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data query method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data query method.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present disclosure is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present disclosure. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for this description.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and its practical application. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of data query, comprising:

2. The data query method of claim 1, before the step of obtaining the data query request is executed, further comprising:

3. The data query method according to claim 2, wherein the determining, according to the writing result, compression index information corresponding to the compressed data includes:

4. The data query method according to claim 3, wherein the creating a compression index corresponding to the compressed data according to the writing result includes:

determining a data type of the compressed data;

5. The data query method according to claim 1, before the steps of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data block included in the target partition are performed, further comprising:

6. The data query method according to claim 5, if the determination result of detecting whether the query attribute information includes the time information is yes, performing the following steps:

7. The data query method according to claim 6, if the determination result of determining whether the time information meets the preset time division condition is negative, performing the following steps:

8. The data query method according to claim 1, wherein, when the query attribute information is not matched with the attribute information, determining a second database having a data migration relationship with the first database, and before the step of reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information is executed, the method further comprises:

9. The data query method according to claim 1, wherein the reading the storage relationship table corresponding to the second database according to the partition key and the query attribute information includes:

10. The data query method of claim 9, the determining index information according to the reading result, comprising:

11. A data query apparatus, comprising:

12. A computing device, comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the method of:

13. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the data query method of any one of claims 1 to 10.