CN111680063A - Method and device for querying data in Elasticissearch paging mode - Google Patents
Method and device for querying data in Elasticissearch paging mode Download PDFInfo
- Publication number
- CN111680063A CN111680063A CN202010448278.0A CN202010448278A CN111680063A CN 111680063 A CN111680063 A CN 111680063A CN 202010448278 A CN202010448278 A CN 202010448278A CN 111680063 A CN111680063 A CN 111680063A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- paging
- page
- querying
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24549—Run-time optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2425—Iterative querying; Query formulation based on the results of a preceding query
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for querying data in an Elasticissearch paging mode, wherein the method comprises the following steps: acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; starting a timing task according to a data query condition, wherein the timing task is used for querying data meeting query fields in each query period in batch from an elastic search at regular time, storing the data into a preset database to obtain a plurality of data blocks, and the data volume contained in each data block does not exceed a preset threshold; receiving a data query request for an appointed paging, and searching a target data block for storing data of the appointed paging from a plurality of data blocks of a preset database according to the data query request for the appointed paging; from the target data block, data specifying pagination is queried. The invention realizes the technical effect of page jump access of the Elasticissearch data under the condition of deep paging.
Description
Technical Field
The invention relates to the field of big data, in particular to a method and a device for querying data in an Elasticissearch paging mode.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the rapid development of information technology, more and more software applications appear, which can generate a large amount of log information in the using process, and can mine a lot of valuable information through the analysis of the log information. Especially, some log information related to user operation behaviors can be mined out to obtain a lot of information with commercial value by analyzing the user operation behavior log.
The ElasticSearch is a distributed, high-expansion and high-real-time search and data analysis engine, and is used for analyzing mass log data. The ElasticSearch is usually used in combination with logstack, which is a log collection and analysis engine and can collect log data, clean and convert the log data, and output the log data to the ElasticSearch.
At present, in the prior art, a paging query mechanism provided by an Elasticsearch is mainly used to implement data query, which mainly includes the following two types:
firstly, adopting a from + size mode to perform paging query, giving the starting position and the paging size of a paging, and returning a data query result; the paging query mode is only suitable for shallow paging, and has good performance when 0-10000 data are queried by from + size; when the query data volume is within 10000-50000 strips, the performance is good, but when the data volume exceeds 50000 strips, the problem of serious performance reduction occurs, and even the elastic search cluster can be dragged down, so that other query and write functions are influenced.
And secondly, paging query in a scroll cursor rolling mode is adopted, and target paging data is obtained through continuous rolling. Although the paging query method can realize deep paging data query, since the scrolll is similar to the cursor in SQL, only one page of content can be obtained at a time by using the scrolll, and then one scrolll _ id is returned. The content of the next page is continuously obtained according to the returned scroll _ id, so that the data query mode of scrolling a scroll cursor is not suitable for the situation with page jump.
Therefore, a paging query method capable of realizing page skipping access to data under deep paging is urgently needed to accelerate the query speed of data in the elastic search.
Disclosure of Invention
The embodiment of the invention provides a method for querying data in an Elasticissearch paging, which is used for solving the technical problem that the data cannot be accessed by skipping pages under deep paging in the conventional Elasticissearch paging data querying mode, and comprises the following steps: acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; starting a timing task according to a data query condition, wherein the timing task is used for querying data meeting query fields in each query period in batch from an elastic search at regular time, storing the data into a preset database to obtain a plurality of data blocks, and the data volume contained in each data block does not exceed a preset threshold; receiving a data query request for an appointed paging, and searching a target data block for storing data of the appointed paging from a plurality of data blocks of a preset database according to the data query request for the appointed paging; from the target data block, data specifying pagination is queried.
An embodiment of the present invention further provides an apparatus for querying data in an Elasticsearch paging, so as to solve a technical problem that a conventional manner for querying data in an Elasticsearch paging cannot implement page skipping access to data in a deep paging, where the apparatus includes: the configuration module is used for acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; the data pre-storage module is used for starting a timing task according to the data query condition, wherein the timing task is used for regularly querying data meeting query fields in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data quantity contained in each data block does not exceed a preset threshold value; the data query module is used for receiving a data query request for a specified page and searching a target data block for storing data of the specified page from a plurality of data blocks of a preset database according to the data query request for the specified page; and the data query module is used for querying the data of the specified pages from the target data block.
The embodiment of the invention also provides computer equipment, which is used for solving the technical problem that the existing method for querying data in the Elasticsearch paging can not realize the data access in the deep paging skipping.
An embodiment of the present invention further provides a computer-readable storage medium, which is used to solve the technical problem that the existing manner of querying data in an Elasticsearch paging cannot realize data access by skipping in a deep paging.
In the embodiment of the invention, a timing task is set according to preset query fields and query cycles, data meeting the query fields in each query cycle are queried in batches from an elastic search, the data are stored in a preset database to obtain a plurality of data blocks, the data volume contained in each data block is enabled not to exceed a preset threshold value, after a data query request for a specified page is received, a target data block for storing the data of the specified page is searched from the data blocks of the preset database according to the data query request for the specified page, and the data of the specified page is queried from the target data block.
According to the embodiment of the invention, data are inquired from the elastic search in batches and stored in the preset database in a block mode, after a data inquiry request for the specified page is received, the data block of the data of the specified page stored in the preset database is searched, and the data of the specified page is inquired from the data block, so that the embodiment of the invention can be suitable for a scene of deep paging and a scene of page skipping to access the data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a flowchart of a method for querying data of an Elasticsearch paging according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating principles of an Elasticsearch paging query data provided in an embodiment of the present invention;
fig. 3 is a schematic diagram of an apparatus for querying data of an Elasticsearch paging according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are used in an open-ended fashion, i.e., to mean including, but not limited to. Reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is for illustrative purposes to illustrate the implementation of the present application, and the sequence of steps is not limited and can be adjusted as needed.
Because the data of the Elasticsearch is changed in real time and the data size is relatively large, when a data requestor (e.g., a certain service system) queries data from the Elasticsearch, it is usually necessary to give a query condition, for example, two parameters, pageSize and pageNum, are given, and a data interface of the Elasticsearch screens out data meeting the paging query condition according to the two parameters and returns the data to the data requestor.
The inventor finds that, in general, the query fields of the service side calling interface are limited, and the relationship among the fields can be exhaustive; the time range of the query data of the calling interface of the service party is fixed, and the starting time and the ending time of the query data are not supported to be arbitrarily specified. Usually, a service party sets a fixed time range according to the requirements of the service party, for example, the time range of querying data each time is set to be an integral point range of each day, that is, data of 14:00 to 15:00 each day, data of 15:00 to 16:00 each day, and the like are queried.
Based on the above inventive concept, an embodiment of the present invention provides a method for querying data of an Elasticsearch paging, and fig. 1 is a flowchart of the method for querying data of the Elasticsearch paging provided in the embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
s101, acquiring a preset data query condition, wherein the data query condition comprises: a query field and a query period.
It should be noted that the query field included in the above S101 may include: one or more fields. In a case that the query field includes a plurality of fields, a combination of the fields may be used as a query condition, and thus, after the above S101, the method for querying data for an Elasticsearch paging provided by the embodiment of the present invention may further include the following steps: generating one or more field combination relations according to each query field contained in the data query condition; creating identification information of each field combination relation; and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation. The data block information of each data block may include any one of: the index information of each data block, the data entry range stored on each data block, the accumulated data entry range corresponding to each data block, and the query time of the first data and the last data on each data block.
Alternatively, when creating the identification information of each field combination relationship, the following steps may be performed: acquiring an expression of each field combination relation; calculating the hash value of each field combination relational expression; and taking the hash value of each field combination relation expression as the identification information of each field combination relation.
For example, in the case where the query field includes two fields, a and B, when a data requester (service) queries data through the two fields, the relationship between the fields may be: four of (a ═ XXX), (B ═ XXX), (a ═ XXX AND B ═ XXX), (a ═ XXX OR B ═ XXX). Therefore, in the embodiment of the present invention, a key-value data structure is generated based on the query field mapping, where key is a field relational expression and value is a code value after hash coding is performed on each relationship. Therefore, the method is mainly used for only storing the code value in the pre-calculation data storage, so that the storage space is compressed, and the retrieval efficiency in the storage is improved.
S102, starting a timing task according to a data query condition, wherein the timing task is used for regularly querying data meeting query fields in each query period in batches from an elastic search, storing the data into a preset database to obtain a plurality of data blocks, and the data volume contained in each data block does not exceed a preset threshold value.
It should be noted that, in order to implement a skip-page access query of the Elasticsearch data under the deep paging, in the embodiment of the present invention, a database is pre-constructed, and the data in the Elasticsearch is read to the database through a timing task, so that after a data query request is received, the data can be queried from the database.
It should be noted that, because the data in the Elasticsearch is read into the preset database at regular time through the timing task, the data query time range included in the data query request needs to be later than the execution time of the timing task, and in an actual service scenario, the execution mode of the timing task may be determined according to the service requirement. For example, the query cycle of the Elasticsearch interface requesting for querying data is one day, that is, data is queried once a day, and the timing task is used for reading the data of the previous day from the Elasticsearch and storing the data into the preset database; if the time range of the data query requested by the Elasticissearch interface is one hour, namely, the data query is performed once every hour, the timing task is used for reading the data of the last hour from the Elasticissearch interface and storing the data into the preset database.
Due to the from + size paging query mode of the Elasticissearch, the Elasticissearch can well support the query efficiency and the paging efficiency when the data size is 0-10000. Therefore, as a preferred implementation manner, the embodiment of the present invention may set that each data block contains no more than 10000 pieces of data, that is, the preset threshold is 10000. By executing the timing task in S102, the data in the Elasticsearch can be segmented into a group according to 10000 pieces, and then stored in a preset database (a database for storing pre-calculation data).
The Elasticissearch data analysis engine can firstly query first 10000 pieces of batch data according to a timing task, record the position of the last piece of data and write the batch of data into a preset database; after the writing is finished, the position of the last recorded data is taken as the starting point, the second 10000 pieces of batch data are continuously inquired, and the like, until the data meeting the conditions are inquired and written into the preset database. Optionally, each data block dataitem in the preset database may contain the following fields:
first query _ code: keeping consistent with the code in the 'inquiry field mapping' in the timing task
Time _ range: keeping consistent with the query time range in the timing task
③ es _ still: the time of the first data in the 10000 data
Es _ etime: the time of the last data in the 10000 data
And fifthly, hit _ num, wherein the number of the obtained items is 10000 when the data volume is sufficient, and the number of the obtained items is 10000 when the data volume is less than 10000.
Sixthly, addup _ num: and when the current data strip data is calculated, the accumulated hit _ num value under the condition is calculated.
Table 1 shows data which are inquired in batches from the Elasticissearch in the inquiry time range of 14: 00-15: 00. As shown in table 1, the query time of the first data and the last data in each data block can be accurate to milliseconds, so that the segmentation result is more accurate.
TABLE 1 batch query results for a certain query period
| query_code | time_range | es_stime | es_etime | hit_num | addup_num | index |
| Code1 | 2019/8/1714:00～15:00 | 14:00:00:000 | 14:22:23:294 | 10000 | 10000 | 1 |
| Code1 | 2019/8/1714:00～15:00 | 14:22:23:600 | 14:40:58:765 | 10000 | 20000 | 2 |
| Code1 | 2019/8/1714:00～15:00 | 14:41:01:654 | 14:59:59:999 | 8000 | 28000 | 3 |
S103, receiving a data query request for the specified page, and searching a target data block for storing the data of the specified page from a plurality of data blocks of a preset database according to the data query request for the specified page.
It should be noted that the data query request may include: the page size and the number of pages of the designated page, S103 may be implemented by: determining a data entry range of the specified paging according to the paging size and the page number contained in the data query request; and traversing each data block in the preset database, and searching the data block for storing the data of the appointed paging according to the data entry range of each data block and the data entry range of the appointed paging.
It should be noted that the query period of the data query request is related to the timing duration of the timing task, and if the timing task reads the data of the last hour from the elastic search every hour, the query period corresponding to the data query request is one hour, for example, querying the data between 14:00 and 15:00, querying the data between 15:00 and 16:00, and the like.
S104, inquiring the data of the appointed paging from the target data block.
As an optional implementation manner, in the above S104, a from + size paging query manner of the Elasticsearch may be adopted, and data of the specified page is queried from the target data block according to the page size and the page number included in the data query request.
As can be seen from the above, in the method for querying data in pages of an Elasticsearch provided in the embodiment of the present invention, a timing task is set according to a preset query field and a preset query cycle, data satisfying the query field in each query cycle is queried in batches from the Elasticsearch, and is stored in a preset database to obtain a plurality of data blocks, so that the data amount included in each data block does not exceed a preset threshold, after a data query request for a specified page is received, and according to the data query request for the specified page, a target data block storing data for the specified page is searched from a plurality of data blocks in the preset database, and the specified data is queried from the target data block. Since the amount of data per data block does not exceed the preset threshold,
according to the method for querying data in the Elasticissearch paging provided by the embodiment of the invention, data is queried from the Elasticissearch in batches and stored in the preset database in a blocking mode, after a data query request for a specified paging is received, the data block of the data of the specified paging stored in the preset database is searched, and the data of the specified paging is queried from the data block, so that the embodiment of the invention can be suitable for a scene of deep paging and a scene of data access by page skipping.
In an optional embodiment, the method for querying data for an Elasticsearch paging provided in the embodiment of the present invention may further include the following steps: and returning the data of the specified page according to the data query request.
Fig. 2 is a schematic diagram of principle of the Elasticsearch paging query data provided in the embodiment of the present invention, as shown in fig. 2, the timing task firstly queries the Elasticsearch to obtain corresponding data according to the query field and the time range, puts the data into the pre-computed data storage, when a query interface request comes, reads the corresponding data from the pre-computed data storage to obtain a pre-computed data result, and requests the Elasticsearch again to obtain a final result.
As shown in fig. 2, an interface request is sent query and time _ range. The query can select a code value after the query is coded according to a key-value key value pair mapped by the query field, wherein time _ range is a time range, pageSize is a page size, and pageNum is a page number.
Calculating addup _ page _ num ═ pageSize × pageNum; and querying the dataitem meeting the conditions in the pre-calculation data storage according to the query _ code and the time _ range, traversing, finding out the first addup _ num > addup _ page _ num data, and taking out the es _ still and the es _ time of the data.
Since the data is already blocked by ten thousand blocks in the pre-calculation data storage, the target paged data can be obtained by paging again in the mode of from + size. In the query request of the structured Elasticsearch, from ═ pageSize (pageNum-1) and size ═ pageSize.
When the query condition requested by the interface side is: the time _ range is 2019/8/1714: 00-15: 00,
query _ Code1, dividing a page according to fifty pieces of data, and needing to jump directly to the 215 th page, the calculation method is as follows:
calculating addup _ page _ num 215 50 10750 according to pageSize 50 and pageNum 215; querying the pre-calculation data storage, finding the dataitem meeting the condition of the first add _ up _ num > addup _ page _ num, and the index of the dataitem is 2, and taking the es _ still and the es _ time of the data, which are 14:22:23:600 and 14:40:58:765 respectively
To acquire data on page 215 using the from + size scheme, from 50 (215-1) +1 10701; size 50, call pseudo code as follows:
hits obtained by the above codes, that is, data specifying a page (target page number), may be returned to the data requester (data caller).
By the method for paging query data of the Elasticissearch, provided by the embodiment of the invention, a user platform for data query can be developed, and the user platform can provide that all data viewed by a user are stored on the Elasticissearch. When a user wants to check data, the user can screen out target data by clicking corresponding selection buttons through some selection buttons provided on the platform, and due to the fact that the data size is large, the target data are displayed in a front-end interface in a paging mode, and the user can jump to the current page number to check the corresponding data by clicking the corresponding page number. Therefore, when the user platform and the elastic search interact with each other for data query, the interface needing to be queried can meet the page jump and page turning under the deep paging. However, according to the previous discussion, the existing scheme can not support the efficiency and the implementation mode well, but the technical scheme of the invention can solve the service pain point well, realize the page jump and page turning function under deep paging, and enable the user to turn pages freely and browse data in all directions.
Based on the same inventive concept, an apparatus for querying data of an Elasticsearch paging is further provided in the embodiments of the present invention, as described in the following embodiments. Because the principle of solving the problem of the embodiment of the device is similar to the method for querying the data in the Elasticsearch paging, the implementation of the embodiment of the device can refer to the implementation of the method, and repeated details are not repeated.
Fig. 3 is a schematic diagram of an apparatus for querying data of an Elasticsearch paging according to an embodiment of the present invention, as shown in fig. 3, the apparatus may include: a configuration module 31, a data pre-storage module 32, a data request module 33 and a data query module 34.
The configuration module 31 is configured to obtain a preset data query condition, where the data query condition includes: a query field and a query period; the data pre-storage module 32 is configured to start a timing task according to a data query condition, where the timing task is configured to query, in batches, data meeting a query field in each query period from an elastic search at a fixed time, and store the data in a preset database to obtain a plurality of data blocks, where a data amount included in each data block does not exceed a preset threshold; the data request module 33 is configured to receive a data query request for a specified page, and search, according to the data query request for the specified page, a target data block storing data of the specified page from a plurality of data blocks in a preset database; and a data query module 34 for querying data of the specified page from the target data block.
As can be seen from the above, in the apparatus for querying data in an Elasticsearch paging provided in the embodiment of the present invention, the configuration module 31 obtains the preset data query condition including the query field and the query period; starting a timing task through a data pre-storage module 32 according to a preset data query condition, querying data meeting query fields in each query period in batch from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, so that the data volume contained in each data block does not exceed a preset threshold; receiving a data query request for a specified page through the data request module 33, and searching a target data block storing data of the specified page from a plurality of data blocks of a preset database according to the data query request for the specified page; finally, the data of the specified page is queried from the target data block through the data query module 34.
According to the device for querying data in the Elasticissearch paging provided by the embodiment of the invention, data is queried from the Elasticissearch in batches and stored in the preset database in a blocking mode, after a data query request for a specified paging is received, the data block of the data of the specified paging stored in the preset database is searched, and the data of the specified paging is queried from the data block, so that the embodiment of the invention can be suitable for a scene of deep paging and a scene of data access by page skipping.
It should be noted that, in the apparatus for querying data in an Elasticsearch paging according to the embodiment of the present invention, the data query request includes: the data request module 33 is further configured to determine a data entry range of the designated page according to the page size and the page number included in the data query request; and traversing each data block in the preset database, and searching the data block for storing the data of the appointed paging according to the data entry range of each data block and the data entry range of the appointed paging.
In an optional implementation, in the apparatus for querying data in an Elasticsearch paging according to the embodiment of the present invention, the data querying module 34 may further be configured to query the data in the specified paging from the target data block according to the paging size and the page number included in the data querying request by using a from + size paging querying manner of the Elasticsearch.
In an optional implementation, the apparatus for querying data for an Elasticsearch paging according to an embodiment of the present invention may further include: the data processing module 35 is configured to generate one or more field combination relationships according to each query field included in the data query condition; creating identification information of each field combination relation; and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation.
Optionally, based on the foregoing embodiment, in the apparatus for querying data for an Elasticsearch paging provided in the embodiment of the present invention, the data block information of each data block includes any one of the following: the index information of each data block, the data entry range stored on each data block, the accumulated data entry range corresponding to each data block, and the query time of the first data and the last data on each data block.
As an optional implementation manner, the data processing module 35 is further configured to obtain an expression of each field combination relationship; calculating the hash value of each field combination relational expression; and taking the hash value of each field combination relation expression as the identification information of each field combination relation.
In an optional implementation, the apparatus for querying data for an Elasticsearch paging according to an embodiment of the present invention may further include: and the data output module 36 is configured to return the data of the specified page according to the data query request.
The embodiment of the invention also provides computer equipment, which is used for solving the technical problem that the existing method for querying data in the Elasticsearch paging can not realize the data access in the deep paging skipping.
The embodiment of the invention also provides a computer readable storage medium, which is used for solving the technical problem that the existing mode of the Elasticissearch paging query data cannot realize the data skipping access under the deep paging.
In summary, embodiments of the present invention provide a method, an apparatus, a computer device, and a computer readable storage medium for querying data in an Elasticsearch paging, where the query characteristic of the Elasticsearch is utilized, and the data in the Elasticsearch is predicted to be segmented, so that a page skipping and page turning function of the Elasticsearch data in a deep paging can be implemented in a large data volume, and a user can freely page and browse the data in an all-around manner. The embodiment of the invention improves the query efficiency and the response speed of the Elasticissearch interface.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for querying data in an Elasticissearch paging is characterized by comprising the following steps:
acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period;
starting a timing task according to the data query condition, wherein the timing task is used for regularly querying data meeting the query field in each query period in batches from an elastic search, storing the data into a preset database to obtain a plurality of data blocks, and the data volume contained in each data block does not exceed a preset threshold;
receiving a data query request for a specified page, and searching a target data block for storing data of the specified page from a plurality of data blocks of the preset database according to the data query request for the specified page;
and querying the data of the specified paging from the target data block.
2. The method of claim 1, wherein the data query request comprises: paging size and page number of a designated page, wherein a data query request for the designated page is received, and a target data block for storing data of the designated page is searched from a plurality of data blocks of the preset database according to the data query request for the designated page, and the method comprises the following steps:
determining the data entry range of the specified paging according to the paging size and the page number contained in the data query request;
and traversing each data block in the preset database, and searching the data block storing the data of the specified paging according to the data entry range of each data block and the data entry range of the specified paging.
3. The method of claim 2, wherein querying the specified page of data from the target block of data comprises:
and querying the data of the specified page from the target data block by adopting a from + size paging query mode of the Elasticissearch according to the page size and the page number contained in the data query request.
4. The method of claim 1, wherein after obtaining the preset data query condition, the method further comprises:
generating one or more field combination relations according to each query field contained in the data query condition;
creating identification information of each field combination relation;
and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation.
5. The method of claim 4, wherein the data block information of each data block comprises any one of: the index information of each data block, the data entry range stored on each data block, the accumulated data entry range corresponding to each data block, and the query time of the first data and the last data on each data block.
6. The method of claim 4, wherein creating identification information for each field combination relationship comprises:
acquiring an expression of each field combination relation;
calculating the hash value of each field combination relational expression;
and taking the hash value of each field combination relation expression as the identification information of each field combination relation.
7. The method of any of claims 1 to 6, wherein after querying the page-specific data from the target data block, the method further comprises:
and returning the data of the specified page according to the data query request.
8. An apparatus for querying data of an Elasticsearch paging, comprising:
the configuration module is used for acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period;
the data pre-storage module is used for starting a timing task according to the data query condition, wherein the timing task is used for regularly querying data meeting the query field in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data volume contained in each data block does not exceed a preset threshold;
the data query module is used for receiving a data query request for a specified page and searching a target data block for storing the data of the specified page from a plurality of data blocks of the preset database according to the data query request for the specified page;
and the data query module is used for querying the data of the specified paging from the target data block.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of Elasticsearch paging query data of any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of the Elasticsearch paging query data of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448278.0A CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448278.0A CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111680063A true CN111680063A (en) | 2020-09-18 |
| CN111680063B CN111680063B (en) | 2023-08-18 |
Family
ID=72453224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010448278.0A Active CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680063B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100247A (en) * | 2020-11-12 | 2020-12-18 | 耀方信息技术(上海)有限公司 | Method and system for querying data by using ElasticSearch |
| CN112650779A (en) * | 2021-01-12 | 2021-04-13 | 浪潮云信息技术股份公司 | Cloud auditing method based on ElasticSearch supporting deep page jump query |
| CN113360503A (en) * | 2021-06-18 | 2021-09-07 | 建信金融科技有限责任公司 | Test data tracking method and device for distributed database |
| CN114661770A (en) * | 2022-04-11 | 2022-06-24 | 平安资产管理有限责任公司 | Data paging query method and device, computer equipment and readable storage medium |
| CN115098805A (en) * | 2022-08-25 | 2022-09-23 | 广州市千钧网络科技有限公司 | Method, device, equipment and storage medium for inquiring paging data information |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126731A (en) * | 2016-07-01 | 2016-11-16 | 百势软件(北京)有限公司 | A kind of method and device obtaining Elasticsearch paged data |
| CN109325047A (en) * | 2018-11-22 | 2019-02-12 | 北京明朝万达科技股份有限公司 | A kind of interactive mode ElasticSearch depth paging query method and apparatus |
| CN109492148A (en) * | 2018-11-22 | 2019-03-19 | 北京明朝万达科技股份有限公司 | ElasticSearch paging query method and apparatus based on Redis |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| US20190361914A1 (en) * | 2017-03-15 | 2019-11-28 | Fauna, Inc. | Methods and systems for a database |
| CN110555034A (en) * | 2018-03-28 | 2019-12-10 | 武汉斗鱼网络科技有限公司 | Data query paging method, device, server and medium |
-
2020
- 2020-05-25 CN CN202010448278.0A patent/CN111680063B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126731A (en) * | 2016-07-01 | 2016-11-16 | 百势软件(北京)有限公司 | A kind of method and device obtaining Elasticsearch paged data |
| US20190361914A1 (en) * | 2017-03-15 | 2019-11-28 | Fauna, Inc. | Methods and systems for a database |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110555034A (en) * | 2018-03-28 | 2019-12-10 | 武汉斗鱼网络科技有限公司 | Data query paging method, device, server and medium |
| CN109325047A (en) * | 2018-11-22 | 2019-02-12 | 北京明朝万达科技股份有限公司 | A kind of interactive mode ElasticSearch depth paging query method and apparatus |
| CN109492148A (en) * | 2018-11-22 | 2019-03-19 | 北京明朝万达科技股份有限公司 | ElasticSearch paging query method and apparatus based on Redis |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100247A (en) * | 2020-11-12 | 2020-12-18 | 耀方信息技术(上海)有限公司 | Method and system for querying data by using ElasticSearch |
| CN112650779A (en) * | 2021-01-12 | 2021-04-13 | 浪潮云信息技术股份公司 | Cloud auditing method based on ElasticSearch supporting deep page jump query |
| CN113360503A (en) * | 2021-06-18 | 2021-09-07 | 建信金融科技有限责任公司 | Test data tracking method and device for distributed database |
| CN114661770A (en) * | 2022-04-11 | 2022-06-24 | 平安资产管理有限责任公司 | Data paging query method and device, computer equipment and readable storage medium |
| CN115098805A (en) * | 2022-08-25 | 2022-09-23 | 广州市千钧网络科技有限公司 | Method, device, equipment and storage medium for inquiring paging data information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111680063B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111680063B (en) | Method and device for paging query data by elastic search | |
| CN107247808B (en) | Distributed NewSQL database system and picture data query method | |
| CN107103032B (en) | Mass data paging query method for avoiding global sequencing in distributed environment | |
| US5649181A (en) | Method and apparatus for indexing database columns with bit vectors | |
| US8700605B1 (en) | Estimating rows returned by recursive queries using fanout | |
| CN108549569B (en) | Method and equipment for searching information in application program | |
| US20090248764A1 (en) | Implementing Dynamic Processor Allocation Based Upon Data Density | |
| CN112988863A (en) | Elasticissearch-based efficient search engine method for heterogeneous multiple data sources | |
| CN110795458B (en) | Interactive data analysis method, device, electronic equipment and computer readable storage medium | |
| US9904705B2 (en) | Data table performance optimization | |
| CN108021333B (en) | System, device and method for randomly reading and writing data | |
| CN105488201A (en) | Log inquiry method and system | |
| CN110704336B (en) | Data caching method and device | |
| CN108846021B (en) | Mass small file storage method based on user access preference model | |
| CN103019887A (en) | Data backup method and device | |
| CN103218365A (en) | SS Table file data processing method and system | |
| US20110179013A1 (en) | Search Log Online Analytic Processing | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| US20050027684A1 (en) | Database system and data accessing method thereof | |
| CN116737067A (en) | Storage loading structure and method of graph data | |
| JP2000099540A (en) | Multi-dimensional database managing method | |
| CN110334073A (en) | A kind of metadata forecasting method, device, terminal, server and storage medium | |
| CN106446080B (en) | Data query method, query service equipment, client equipment and data system | |
| IL262576B (en) | System and method for storing and querying document collections | |
| CN114996270A (en) | Method and device for inquiring paging data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |