WO2017005094A1

WO2017005094A1 - Data query method and device

Info

Publication number: WO2017005094A1
Application number: PCT/CN2016/086870
Authority: WO
Inventors: 林锦成
Original assignee: 阿里巴巴集团控股有限公司; 林锦成
Priority date: 2015-07-03
Filing date: 2016-06-23
Publication date: 2017-01-12
Also published as: CN106326309B; CN106326309A

Abstract

Disclosed are a data query method and device. The data query method comprises: receiving a data query request, with the data query request including information about data to be queried (101); determining a single-page data identifier corresponding to the information about data to be queried according to data page information about a database acquired in advance, with the data page information at least including every single-page data identifier in the database (102); requesting data corresponding to the single-page data identifier corresponding to the information about data to be queried from the database (103); and querying data to be queried corresponding to the information about data to be queried in the requested data (104). By acquiring data page information by utilizing a single-page data identifier in a database, and according to the data page information, data can be queried by directly acquiring one or more pages of data corresponding to information about data to be queried in the database. The query method simplifies the data query process, and increases the efficiency of data query.

Description

Data query method and device

Technical field

The present application relates to the field of data processing technologies, and in particular, to a data query method and a data query device.

Background technique

A database is a repository that organizes, stores, and manages data according to its data structure. With the development of information technology and the market, data management is no longer just about storing and managing data, but transforming into a variety of data management methods that users need. There are many types of databases, ranging from the simplest tables that store various data to the large database systems that can store massive amounts of data. With the wide application of large databases, how to quickly query data in the database is particularly important.

In a structured database like mysql, since the primary key is basically sequential, it is easy to query the data based on the primary key. In the semi-structured storage environment of HBASE, data is stored in the form of key-value, rowkey is the key in the storage of HBASE, analogous to the primary key in mysql, usually the user uses rowkey as the query field. In HBASE, in order to achieve data balance distribution, generally rowkey does not use self-added value, but uses a hash value, that is, rowKey is irregular. Therefore, in this case, when data query is performed based on rowkey, only one page of data in HBASE can be searched sequentially. This kind of query method, especially in the case of querying a large amount of data, takes a long time and is inefficient.

Therefore, a technical problem that needs to be solved urgently by those skilled in the art is how to improve the efficiency of data query.

Summary of the invention

The technical problem to be solved by the embodiments of the present application is to provide a data query method, which can improve data query efficiency.

Correspondingly, the embodiment of the present application further provides a data query device for ensuring implementation and application of the foregoing method.

In order to solve the above problem, the present application discloses a data query method, including:

Receiving a data query request, where the data query request includes data information to be queried;

Determining, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the to-be-queried data information; the data page information includes at least each single page data identifier in the database;

Requesting, by the database, data corresponding to the single page data identifier corresponding to the to-be-queried data information;

The data to be queried corresponding to the data to be queried is queried in the requested data.

Further, a method for obtaining data page information of the database includes:

Sending a data page information query request to the database;

The data page information is received and added to the memory queue, the data page information including at least a single page data identifier of each page of the database.

Further, the to-be-queried data information includes at least one rowkey of the data to be queried.

Further, the to-be-queried data information includes a start and stop page of the data to be queried in the database; the data page information further includes page code information corresponding to the single-page data identifier of each page of the database.

Further, the to-be-queried data information includes a start page and a data amount of the data to be queried in the database; the data page information further includes page code information and a page corresponding to the single-page data identifier of each page of the database. The amount of data inside.

Further, when the determined single page data identifier corresponding to the to-be-queried data information is at least two, the requesting, by the database, data corresponding to the single page data identifier corresponding to the to-be-queried data information, include:

Enabling at least two threads to pull different single page data identifiers corresponding to the to-be-queried data information in the memory queue in parallel;

Occupying the at least two threads to request data corresponding to the pulled single page data identifier to the database in parallel.

Further, the database is a semi-structured storage database with an irregular key value.

The embodiment of the present application further discloses a data query device, including:

a request receiving unit, configured to receive a data query request, where the data query request includes data information to be queried;

a determining unit, configured to determine, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the to-be-queried data information; the data page information includes at least each single page data identifier in the database ;

a data requesting unit, configured to request, from the database, data corresponding to the single-page data identifier corresponding to the to-be-queried data information;

The query unit is configured to query, in the requested data, the data to be queried corresponding to the data information to be queried.

Further, the device further includes:

An information obtaining unit configured to obtain data page information of the database;

The information acquiring unit includes:

a request subunit configured to send a data page information query request to the database;

The storage subunit is configured to receive the data page information and add it to the memory queue, the data page information including at least a single page data identifier of each page of the database.

Further, when the determining unit determines at least two single page data identifiers corresponding to the to-be-queried data information, the data requesting unit includes:

Pulling a subunit, configured to enable at least two threads to pull different single page data identifiers corresponding to the to-be-queried data information in the memory queue in parallel;

The data requesting subunit is configured to occupy the data corresponding to the at least two threads in parallel requesting the pulled single page data identifier from the database.

Compared with the prior art, the embodiments of the present application include the following advantages:

The embodiment of the present application obtains the data page information by using the single page data identifier in the database, and according to the data page information, can directly obtain a page or a certain page of data corresponding to the data information to be queried in the database for data query, The query method does not need to perform data query page by page in the database, which simplifies the data query process and improves the data query efficiency.

DRAWINGS

1 is a flow chart showing the steps of an embodiment of a data query method of the present application;

2 is a flow chart showing the steps of an embodiment of a method for obtaining data page information of a database in the present application;

3 is a flow chart showing the steps of another data query method embodiment of the present application;

4 is a flow chart showing the steps of another embodiment of the data query method of the present application;

5 is a flow chart showing the steps of another data query method embodiment of the present application;

6 is a structural block diagram of an embodiment of a data query apparatus of the present application;

7 is a structural block diagram of another embodiment of a data query apparatus of the present application;

8 is a structural block diagram of an information acquisition unit in the present application;

9 is a structural block diagram of a data request unit in the present application.

detailed description

The above described objects, features and advantages of the present application will become more apparent and understood.

Referring to FIG. 1 , a flow chart of steps of an embodiment of a data query method of the present application is shown, which may specifically include the following steps:

Step 101: Receive a data query request, where the data query request includes data information to be queried.

In the embodiment of the present application, the device for querying data (hereinafter referred to as the device) obtains the data to be queried information included in the request after receiving the data query request, and the data to be queried may include at least one data to be queried. The rowkey needs the device to find the data to be queried corresponding to the rowkey in the database; the data information to be queried may also include the start and end pages of the data to be queried in the database, and the device needs to find the start and end page in the database. The data to be queried may also be a start page and a data amount including the data to be queried in the database, and the device needs to satisfy the data amount after the start page is found in the database. Of course, the data to be queried may also include other information for querying data, which will not be enumerated here.

Step 102: Determine, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the data information to be queried, where the data page information includes at least each single page data identifier in the database.

The device obtains the data page information of the database in advance before performing the step, which may be the device requesting the database in advance. The data page information includes at least each order in the database The page data identifier, the single page data identifier can uniquely identify the corresponding data page, for example, in the HBASE database, the marker is the starting identifier of the data page; the data page information can also include a single page with each page in the database. The page data identifies corresponding page number information; it may also include an amount of in-page data corresponding to a single page data identifier of each page of data in the database.

After obtaining the data page information, according to the content included in the data information to be queried, a single page data identifier corresponding to the data information to be queried may be determined, and the determined single page data identifier may be one or multiple The data pages corresponding to multiple single page data identifiers may be continuous or discontinuous in the database.

Step 103: Request data corresponding to the single page data identifier corresponding to the data information to be queried to the database.

When the determined single page data identifier is one, the thread may be required to request the data corresponding to the single page data identifier from the database, and when the determined single page data identifier is multiple, the thread may be occupied in parallel to the database. Request data corresponding to each single page data identifier.

Step 104: Query the data to be queried corresponding to the data information to be queried in the requested data.

After requesting the data, the data to be queried can be queried according to the data information to be queried. The data to be queried may be the data requested in the previous step, or may be part of the data in the requested data.

In another embodiment of the present application, as shown in FIG. 2, the method for obtaining data page information of a database may include the following steps:

Step 201: Send a data page information query request to the database.

The device sends a query request to the database. After receiving the request, the database obtains at least a single page data identifier of each page of data according to the request, and may also obtain page number information and/or amount of data in the page corresponding to the single page data identifier of each page of data in the database according to the requested content. Generate data page information, then Then return it to the device.

Step 202: Receive data page information, and add data page information to a memory queue, where the data page information includes at least a single page data identifier of each page of data in the database.

After receiving the data page information sent by the data, the device adds it to the memory queue so that it can be quickly called at any time.

By obtaining the data page information, the device can quickly and easily determine a page or a certain number of pages in the database for data query according to the data query request, and does not simplify the data query process and improve the data query efficiency.

The following describes the data query methods in different application scenarios.

Referring to FIG. 3, a flow chart of steps of another data query method embodiment of the present application is shown, which may specifically include the following steps:

Step 301: Receive a data query request, where the data query request includes at least one rowkey of the data to be queried.

The application scenario of this embodiment is to perform a query according to the rowkey of the data.

Step 302: Determine, according to the data page information of the database obtained in advance, all the single page data identifiers of the database.

In this case, the data page information of the database obtained in advance may include only a single page data identifier of each page of data in the database. In this embodiment, since a specific data is to be queried, instead of a specified data page, all data pages in the database need to be queried, and the single page data identified in this step is identified as all single page data of the database. Logo.

In step 303, at least two threads are enabled to pull all the different single page data identifiers in the memory queue in parallel.

In this step, multiple threads need to be enabled, and multiple threads execute in parallel. Each thread pulls a single page data identifier in the memory queue, and the single page data identifiers drawn by different threads are different.

Step 304: The at least two threads are occupied in parallel to request data corresponding to the pulled single page data identifier from the database.

In the above step, after each thread pulls the single page data identifier, the thread requests the data corresponding to the pulled single page data identifier in parallel.

Step 305: Query, in the requested data, the data to be queried corresponding to the rowkey of the data to be queried.

In the above step, each thread inquires the data corresponding to the extracted single page data identifier, and queries the data of the request path in parallel with the data to be queried corresponding to rowkey in step 301.

The above threads execute in parallel. When a thread query ends, the thread can repeatedly execute steps 303-305 until all the single page data identifiers in the database are pulled.

In this embodiment, the manner in which each data page is retrieved in parallel for data query is compared with the sequential page-by-page query in the prior art, which greatly improves the data query efficiency.

Referring to FIG. 4, a flow chart of steps of another embodiment of the data query method of the present application is shown, which may specifically include the following steps:

Step 401: Receive a data query request, where the data query request includes a start and end page of the data to be queried in the database.

The application scenario of this embodiment is to query data between the start and end pages of a specified data page in the database. The start and end page can also be the same page, that is, requesting a page of data in the database.

Step 402: Determine, according to the data page information of the database obtained in advance, a single page data identifier of the data page between the start and end pages in the database, where the data page information includes a single page data identifier of each page of the database and corresponding Page number information.

In this step, the device may determine a single page data identifier of the data page between the start and end pages according to the single page data identifier of each page of data in the database and its corresponding page number information.

Step 403, enabling at least two threads to pull the determined different single page data identifiers in the memory queue in parallel.

If the determined single page data identifier has only one page, the thread is directly enabled to pull the single page data identifier from the memory queue, and the data corresponding to the single page data identifier is requested from the database.

In this embodiment, when it is determined that there are multiple single page data identifiers, at least two threads are enabled to respectively extract different single page data identifiers in the memory queue.

The process of enabling multiple threads to pull a single page data identification is similar to step 303 in the previous example.

Step 404, occupying the at least two threads to request data corresponding to the extracted single page data identifier in parallel to the database.

This process is similar to step 304 in the foregoing embodiment, and details are not described herein again.

In this embodiment, the data requested in step 404 is the data between the start and end pages of the database requested in step 401.

In this embodiment, the single-page data identifier is used to request corresponding data from the database, which simplifies the operation process of the database, and can obtain data simultaneously through multiple threads, thereby improving data query efficiency.

Referring to FIG. 5, a flow chart of steps of another data query method embodiment of the present application is shown, which may specifically include the following steps:

Step 501: Receive a data query request, where the data query request includes a start page and a data amount of the data to be queried in the database.

The application scenario in this embodiment is that the device needs to query data that reaches the required data amount after the start page.

Step 502: Determine, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the start page and the data amount in the database, where the data page information includes a single page data identifier of each page of the database and Corresponding page number information and amount of data in the page.

In this step, the single page data identifier corresponding to the start page is determined according to the page number information corresponding to the single page data identifier, and then the data after the start page is estimated according to the amount of data in the page corresponding to the single page data identifier and the data amount included in the request. Pages, and determine the single page data identifier corresponding to these data pages. It is possible that the amount of data contained in the request corresponds to all the data in the determined data page, and it is also possible that the amount of data contained in the request is smaller than all the data in the determined data page, for example, in the last data page determined. Need to query some of the data.

Step 503, enabling at least two threads to pull the determined different single page data identifiers in the memory queue in parallel.

Step 504, occupying the at least two threads to request data corresponding to the pulled single page data identifier in parallel to the database.

After the single-page data identifier is determined, the steps 503-504 are similar to the steps 403-404 in the foregoing embodiment, and details are not described herein again.

Step 505: Query, in the requested data, the data to be queried corresponding to the start page and the data amount of the data to be queried in the database.

When the amount of data contained in the request corresponds to all the data in the determined data page, the data requested in step 504 is the data requested in step 501. When the amount of data included in the request is less than all the data in the determined data page, for example, only part of the data in the last data page determined, the data to be queried can be obtained according to the data volume in the request. .

After the data to be queried is obtained according to the above embodiment, the data processing may be further performed according to the obtained data, for example, data comparison, data backup, and the like.

The foregoing method embodiment can be applied to a semi-structured storage database whose database has an irregular key value, such as HBASE, wherein the single page data identifier is a marker.

It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present application are not limited by the described action sequence, because In accordance with embodiments of the present application, certain steps may be performed in other sequences or concurrently. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required in the embodiments of the present application.

Referring to FIG. 6, a structural block diagram of an embodiment of a data query apparatus according to the present application is shown, which may specifically include the following units:

The request receiving unit 601 is configured to receive a data query request, where the data query request includes data information to be queried.

The determining unit 602 is configured to determine, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the to-be-queried data information; the data page information includes at least each single page data in the database Logo.

The data requesting unit 603 is configured to request, from the database, data corresponding to the single page data identifier corresponding to the to-be-queried data information.

The querying unit 604 is configured to query, in the requested data, the data to be queried corresponding to the data information to be queried.

The device obtains the data page information by using the single page data identifier in the database through the above unit, and according to the data page information, directly obtains a certain one of the database corresponding to the data information to be queried. The page or a few pages of data for data query, the device query mode does not need to query the data page by page in the database, which simplifies the data query process and improves the data query efficiency.

In another embodiment, as shown in FIG. 7, the device may further include:

The information acquisition unit 701 is configured to obtain data page information of the database.

The information obtaining unit 701, as shown in FIG. 8, may further include:

The request subunit 801 is configured to send a data page information query request to the database.

The storage sub-unit 802 is configured to receive the data page information and add it to the memory queue, the data page information including at least a single-page data identifier of each page of the database.

In another embodiment, the to-be-queried data information may include at least one rowkey of the data to be queried.

In another embodiment, the to-be-queried data information may further include a start and stop page of the data to be queried in the database; the data page information further includes page code information corresponding to the single-page data identifier of each page of the database.

In another embodiment, the to-be-queried data information may further include a start page and a data amount of the data to be queried in the database; the data page information further includes a single-page data identifier corresponding to each page of the database. Page number information and the amount of data in the page.

In another embodiment, when the determining unit 602 determines that the single page data identifier corresponding to the to-be-queried data information is at least two, as shown in FIG. 9, the data requesting unit 603 may include:

The pull subunit 901 is configured to enable at least two threads to pull different single page data identifiers corresponding to the to-be-queried data information in the memory queue in parallel.

The data requesting subunit 902 is configured to occupy the data corresponding to the at least two threads in parallel requesting the extracted single page data identifier from the database.

The above database may be a semi-structured storage database with an irregular key value, such as HBASE, wherein the single page data identifier is a marker.

The embodiment of the present application further provides an electronic device, including a memory, a transceiver, and a processor.

The processor and the memory and the transceiver are connected to each other through a bus; the bus may be an ISA bus, a PCI bus or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

Wherein, the memory is used to store a program, in particular, the program may include program code, The program code includes computer operating instructions. The memory may include a high speed RAM memory and may also include a non-volatile memory such as at least one disk memory.

The transceiver is configured to receive a data query request, where the data query request includes data to be queried; and request, by the database, data corresponding to the single page data identifier corresponding to the to-be-queried data information.

The processor is used to read the program code in the memory and perform the following steps:

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments can be referred to each other.

Those skilled in the art will appreciate that embodiments of the embodiments of the present application can be provided as a method, apparatus, or computer program product. Therefore, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium. Computer readable media includes both permanent and non-permanent, removable and non-removable media Information storage can be accomplished by any method or technique. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-persistent computer readable media, such as modulated data signals and carrier waves.

Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device Means are provided for implementing the functions specified in one or more of the flow or in one or more blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The instruction device implements the functions specified in one or more blocks of the flowchart or in a flow or block of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device such that a series of operational steps are performed on the computer or other programmable terminal device to produce computer-implemented processing, such that the computer or other programmable terminal device The instructions executed above provide steps for implementing the functions specified in one or more blocks of the flowchart or in a block or blocks of the flowchart.

Although a preferred embodiment of the embodiments of the present application has been described, one of ordinary skill in the art once Additional changes and modifications to these embodiments can be made upon the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the modifications and the modifications

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Furthermore, the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, or terminal device that includes a plurality of elements includes not only those elements but also Other elements that are included, or include elements inherent to such a process, method, article, or terminal device. An element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that comprises the element, without further limitation.

The above is a detailed description of a data query method and a data query device provided by the present application. The principles and implementation manners of the present application are described in the specific examples. The description of the above embodiments is only for help. The method of the present application and its core idea are understood; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and the scope of application. It should be understood that the limitations of the application.

Claims

A data query method, comprising:

Receiving a data query request, where the data query request includes data information to be queried;

Determining, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the to-be-queried data information; the data page information includes at least each single page data identifier in the database;

Requesting, by the database, data corresponding to the single page data identifier corresponding to the to-be-queried data information;

The data to be queried corresponding to the data to be queried is queried in the requested data.
The method of claim 1, wherein the method of obtaining data page information of the database comprises:

Sending a data page information query request to the database;

The data page information is received and added to the memory queue, the data page information including at least a single page data identifier of each page of the database.
The method according to claim 2, wherein the data to be queried includes at least one rowkey of data to be queried.
The method according to claim 2, wherein the data to be queried includes a start and stop page of the data to be queried in the database; and the data page information further includes a single page of data with each page of the database. Identifies the corresponding page number information.
The method according to claim 2, wherein the data to be queried includes a start page and a data amount of the data to be queried in the database; and the data page information further includes data of each page in the database. The single page data identifies the corresponding page number information and the amount of data in the page.
The method according to any one of claims 2 to 5, wherein when the determined single page data identifier corresponding to the to-be-queried data information is at least two, the requesting to the database The data corresponding to the single page data identifier corresponding to the data information to be queried includes:

Enabling at least two threads to pull data information to be queried in the memory queue in parallel Corresponding different single page data identifiers;

Occupying the at least two threads to request data corresponding to the pulled single page data identifier to the database in parallel.
The method according to any one of claims 1 to 5, wherein the database is a semi-structured storage database with an irregular key value.
A data query device, comprising:

a request receiving unit, configured to receive a data query request, where the data query request includes data information to be queried;

a determining unit, configured to determine, according to the data page information of the database obtained in advance, a single page data identifier corresponding to the to-be-queried data information; the data page information includes at least each single page data identifier in the database ;

a data requesting unit, configured to request, from the database, data corresponding to the single-page data identifier corresponding to the to-be-queried data information;

The query unit is configured to query, in the requested data, the data to be queried corresponding to the data information to be queried.
The device according to claim 8, wherein the device further comprises:

An information obtaining unit configured to obtain data page information of the database;

The information acquiring unit includes:

a request subunit configured to send a data page information query request to the database;

The storage subunit is configured to receive the data page information and add it to the memory queue, the data page information including at least a single page data identifier of each page of the database.
The apparatus according to claim 8 or 9, wherein when the determining unit determines at least two single page data identifiers corresponding to the to-be-queried data information, the data requesting unit includes:

Pulling a subunit, configured to enable at least two threads to pull different single page data identifiers corresponding to the to-be-queried data information in the memory queue in parallel;

The data requesting subunit is configured to occupy the data corresponding to the at least two threads in parallel requesting the pulled single page data identifier from the database.