CN111680063B - Method and device for paging query data by elastic search - Google Patents
Method and device for paging query data by elastic search Download PDFInfo
- Publication number
- CN111680063B CN111680063B CN202010448278.0A CN202010448278A CN111680063B CN 111680063 B CN111680063 B CN 111680063B CN 202010448278 A CN202010448278 A CN 202010448278A CN 111680063 B CN111680063 B CN 111680063B
- Authority
- CN
- China
- Prior art keywords
- data
- query
- paging
- block
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24549—Run-time optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2425—Iterative querying; Query formulation based on the results of a preceding query
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a method and a device for paging query data of an elastic search, wherein the method comprises the following steps: acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; starting a timing task according to the data query condition, wherein the timing task is used for periodically querying data meeting query fields in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data amount contained in each data block does not exceed a preset threshold value; receiving a data query request for the specified paging, and searching a target data block for storing the data of the specified paging from a plurality of data blocks of a preset database according to the data query request for the specified paging; from the target data block, the data specifying the page is queried. The application realizes the technical effect of page-skipping access of the elastic search data under the condition of deep paging.
Description
Technical Field
The application relates to the field of big data, in particular to a method and a device for paging query data by an elastic search.
Background
This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the rapid development of information technology, more and more software applications are presented, which generate a large amount of log information in the use process, and through analysis of the log information, a great amount of valuable information can be mined. In particular, some log information related to user operation behaviors can mine out a lot of commercially valuable information by analyzing the user operation behavior log.
The elastiscearch is a distributed, high-expansion and high-real-time search and data analysis engine, and is used for analyzing massive log data. The elastosearch is usually used in combination with a logstack, which is a log collection and analysis engine capable of collecting log data, cleaning and converting the log data, and outputting the log data to the elastosearch.
Currently, the prior art mainly adopts a paging query mechanism provided by an elastic search to realize data query, and mainly comprises the following two types:
firstly, adopting a from+size mode to query the pages, giving the starting position and the page size of the pages, and returning a data query result; the paging query mode is only suitable for shallow paging, and has good performance when the from+size query is performed on 0-10000 pieces of data; when the data amount of query is within 10000-50000 pieces, the performance is good, but when the data amount exceeds 50000 pieces, the problem of serious performance degradation can occur, and even an elastic search cluster can be dragged down, so that other query and writing functions are affected.
Second, the page inquiry of the scroll cursor rolling mode is adopted, and the data of the target page is obtained through continuous rolling. In this paging query mode, although deep paging data query can be realized, since the script is similar to the cursor in SQL, only one page of content can be acquired at a time by using the script, and then one script_id is returned. The content of the next page is continuously acquired according to the returned scroll_id, so that the data query mode of scroll cursor scrolling is not suitable for the scene with jump pages.
Therefore, a paging query mode capable of realizing page-skipping access data under deep paging is urgently needed to accelerate the query speed of data in the elastic search.
Disclosure of Invention
The embodiment of the application provides a method for inquiring data by an elastic search page, which is used for solving the technical problem that the prior method for inquiring data by the elastic search page can not realize data access by a deep-page down-jump page, and comprises the following steps: acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; starting a timing task according to the data query condition, wherein the timing task is used for periodically querying data meeting query fields in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data amount contained in each data block does not exceed a preset threshold value; receiving a data query request for the specified paging, and searching a target data block for storing the data of the specified paging from a plurality of data blocks of a preset database according to the data query request for the specified paging; from the target data block, the data specifying the page is queried.
The embodiment of the application also provides a device for searching data by the elastic search page, which is used for solving the technical problem that the prior method for searching data by the elastic search page can not realize the access of data by the deep page down-jump page, and comprises the following steps: the configuration module is used for acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period; the data pre-storage module is used for starting a timing task according to the data query condition, wherein the timing task is used for inquiring data meeting the query field in each query period in batches from the elastic search and storing the data into a preset database to obtain a plurality of data blocks, and the data amount contained in each data block does not exceed a preset threshold value; the data request module is used for receiving a data query request for the specified paging and searching a target data block for storing the data of the specified paging from a plurality of data blocks of a preset database according to the data query request for the specified paging; and the data query module is used for querying the data of the designated paging from the target data block.
The embodiment of the application also provides a computer device, which is used for solving the technical problem that the prior method for searching the data by paging the data by the elastic search cannot realize the access of the data by the deep page down-jump.
The embodiment of the application also provides a computer readable storage medium, which is used for solving the technical problem that the prior method for inquiring data by the elastic search paging can not realize the access of data by the deep paging.
In the embodiment of the application, a timing task is set according to a preset query field and a preset query period, data meeting the query field in each query period is queried in batches from an elastic search, the data is stored in a preset database, a plurality of data blocks are obtained, the data volume contained in each data block does not exceed a preset threshold value, after a data query request for specified paging is received, and according to the data query request for the specified paging, a target data block for storing the data for the specified paging is searched from the plurality of data blocks in the preset database, and the data for the specified paging is queried from the target data block.
According to the embodiment of the application, the data are queried from the elastic search in batches and stored in the preset database in the blocking way, and when the data query request for the specified paging is received, the data block of the specified paging data stored in the preset database is searched first, and the specified paging data are queried from the data block, so that the embodiment of the application can be suitable for a deep paging scene and a page-skipping access data scene.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a flowchart of a method for paging query data by an elastic search in an embodiment of the present application;
FIG. 2 is a schematic diagram of an elastic search paging query data according to an embodiment of the present application;
fig. 3 is a schematic diagram of an apparatus for paging query data by an elastic search according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present application and their descriptions herein are for the purpose of explaining the present application, but are not to be construed as limiting the application.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are open-ended terms, meaning including, but not limited to. The description of the reference terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The order of steps involved in the embodiments is illustrative of the practice of the application, and is not limited and may be suitably modified as desired.
Since the data of the elastic search is changed in real time and the data size is relatively large, when the data requester (e.g., a certain service system) queries the data from the elastic search, it is generally required to set two parameters, for example, a page size and a page number, according to which the data interface of the elastic search screens out the data meeting the page query condition and returns the data to the data requester.
The inventor finds that in general, the query field of the service party call interface is limited, and the relation among the fields can be exhausted; the time range of the query data of the calling interface of the service party is fixed, and the starting time and the ending time of the query data are not supported to be arbitrarily designated. Typically, the business side sets a fixed time range according to the needs of the business side, for example, sets the time range of each query of data to be the whole daily range, i.e. queries data of 14:00-15:00 each day, data of 15:00-16:00 each day, etc.
Based on the above-mentioned inventive concept, the embodiment of the present application provides a method for paging query data by using an elastic search, and fig. 1 is a flowchart of a method for paging query data by using an elastic search, as shown in fig. 1, where the method includes the following steps:
s101, acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period.
Note that, the query field included in S101 may include: one or more fields. In the case that the query field includes a plurality of fields, the combination of the fields may be used as a query condition, so after S101, the method for paging query data by using elastic search according to the embodiment of the present application may further include the following steps: generating one or more field combination relations according to each query field contained in the data query condition; creating identification information of each field combination relation; and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation. The data block information of each data block may include any one of the following: index information of each data block, a data entry range stored on each data block, an accumulated data entry range corresponding to each data block, and query time of first data and last data on each data block.
Alternatively, when creating the identification information of each field combination relationship, it may be implemented by the following steps: acquiring an expression of each field combination relation; calculating a hash value of each field combination relation expression; and taking the hash value of each field combination relation expression as identification information of each field combination relation.
For example, in the case where the query field includes two fields, a and B, when the data requester (service party) queries data through the two fields, the relationship between the fields may be: (a=xxx), (b=xxx), (a=xxx AND b=xxx), (a=xxx OR b=xxx). Thus, in the embodiment of the application, a key-value data structure is generated based on query field mapping, wherein the key is a field relation expression, and the value is a code value after hash coding is performed on each relation. The method is mainly used for only storing the code value in pre-calculation data storage, so that the storage space is compressed, and the retrieval efficiency in storage is improved.
S102, starting a timing task according to data query conditions, wherein the timing task is used for querying data meeting query fields in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data amount contained in each data block does not exceed a preset threshold value.
It should be noted that, in order to implement page skip access query of the elastic search data under deep paging, the embodiment of the present application constructs a database in advance, and reads the data in the elastic search to the database through a timing task, so that after receiving a data query request, the data can be queried from the database.
It should be noted that, because the data in the elastic search is read to the preset database at regular time through the timing task, the data query time range included in the data query request needs to be later than the execution time of the timing task, and in the actual service scenario, the execution mode of the timing task can be determined according to the service requirement. For example, if the query cycle of the elastic search interface for querying data is one day, that is, data is queried once a day, the timing task is used for reading the data of the previous day from the elastic search and storing the data into the preset database; if the time range of the data query request by the elastic search interface is one hour, namely, data is queried every hour, a timing task is used for reading the data of the last hour from the elastic search and storing the data in a preset database.
Because of the from+size paging query mode of the elastic search, the elastic search can be well supported in the query efficiency and the paging efficiency of 0-10000 data sizes. Thus, as a preferred embodiment, the embodiment of the present application may set the data amount contained in each data block to be not more than 10000, that is, the above-mentioned preset threshold value is 10000. By executing the timing task in S102, the data in the elastic search can be segmented into 10000 pieces, and then stored into a preset database (one database for pre-calculation data storage).
The elastic search data analysis engine can firstly inquire batch data of the first 10000 pieces according to a timing task, record the position of the last piece of data and write the batch of data into a preset database; after the writing is finished, the position of the last recorded data is taken as a starting point, the second 10000 batches of data are continuously inquired, and the like until the data meeting the conditions are all inquired and written into a preset database. Optionally, each data block dataitem in the preset database may contain the following fields:
(1) query_code: consistent with code in "query field map" in timing tasks
(2) time_range: consistent with the query time range in the timing task
(3) es_stine: the time of the first data in the 10000 data at present
(4) es_etime: the current time of the last data in the 10000 data
(5) hit_num is 10000 when the data amount is sufficient, and is 10000 when the data amount is less than 10000, how many pieces are searched out.
(6) addup_num: when the current data piece data is calculated, the accumulated hit_num value under the condition is calculated.
Table 1 shows data from the elastic search in batches with a query time range of 14:00 to 15:00. As shown in table 1, the query time of the first data and the last data in each data block may be accurate to millisecond, so that the segmentation result is more accurate.
Table 1 batch query results for a certain query period
| query_code | time_range | es_stime | es_etime | hit_num | addup_num | index |
| Code1 | 2019/8/1714:00～15:00 | 14:00:00:000 | 14:22:23:294 | 10000 | 10000 | 1 |
| Code1 | 2019/8/1714:00～15:00 | 14:22:23:600 | 14:40:58:765 | 10000 | 20000 | 2 |
| Code1 | 2019/8/1714:00～15:00 | 14:41:01:654 | 14:59:59:999 | 8000 | 28000 | 3 |
S103, receiving a data query request for the specified page, and searching a target data block for storing the data of the specified page from a plurality of data blocks of a preset database according to the data query request for the specified page.
It should be noted that, the data query request may include: the page size and the number of page codes of the specified page, S103 can be implemented by: determining a data entry range of the designated paging according to the paging size and the page number contained in the data query request; traversing each data block in the preset database, and searching the data block for storing the data of the designated paging according to the data entry range of each data block and the data entry range of the designated paging.
It should be noted that, the query period of the data query request is related to the timing duration of the timing task, and if the timing task is reading data of one hour from the elastic search every hour, the query period corresponding to the request for querying the data is one hour, for example, querying data between 14:00 and 15:00, querying data between 15:00 and 16:00, and the like.
S104, inquiring the data of the appointed paging from the target data block.
As an alternative implementation manner, S104 may use the from+size paging query mode of the elastic search, and query the data of the specified page from the target data block according to the page size and the page number included in the data query request.
As can be seen from the foregoing, in the method for querying data in paging of an elastic search provided in the embodiment of the present application, a timing task is set according to a preset query field and a preset query period, data satisfying the query field in each query period is queried in batches from the elastic search, and the data is stored in a preset database to obtain a plurality of data blocks, so that the data amount contained in each data block does not exceed a preset threshold value, after a data query request for a specified page is received, and according to the data query request for the specified page, a target data block storing the data for the specified page is searched from the plurality of data blocks in the preset database, and the data for the specified page is queried from the target data block. Since the data amount of each data block does not exceed the preset threshold,
according to the method for inquiring data in the elastic search page, which is provided by the embodiment of the application, data are inquired from the elastic search in batches and stored in the preset database in a blocking mode, when a data inquiry request for appointed paging is received, the preset database is searched for storing a data block of the appointed paging data, and the appointed paging data are inquired from the data block, so that the method and the device are applicable to a scene of deep paging and a scene of page skip access data.
In an alternative embodiment, the method for paging query data by elastic search provided by the embodiment of the present application may further include the following steps: and returning the data of the designated paging according to the data query request.
Fig. 2 is a schematic diagram of an elastic search paging query data provided in an embodiment of the present application, as shown in fig. 2, a timing task firstly searches for corresponding data by going to an elastic search according to a query field and a time range, puts the data in a pre-calculation data storage, reads the corresponding data from the pre-calculation data storage when a query interface request arrives, obtains a pre-calculation data result, and requests an elastic search to retrieve again, thereby obtaining a final result.
As shown in fig. 2, the interface request will send a query and a time_range. The query can select the code value after the query codes according to the key-value key value pair mapped by the query field, the time_range is the time range, the pageSize is the paging size, and the pageNum page code number.
Calculating addup_page_num= (pagesize×pagenum); and then according to the query_code and the time_range, inquiring the dataitem meeting the conditions in the pre-calculated data storage, traversing, finding out the data of the first addup_num > =addup_page_num, and taking out the es_stine and the es_etime of the data.
Since the data is already segmented into ten thousand pieces in the pre-computed data storage, the data of the target page can be obtained by using the mode of from+size at this time for paging. In the constructed elastic search query request, from=pagesize (pageNum-1), size=pagesize.
When the query conditions requested by the interface side are as follows: time range is 2019/8/1714:00-15:00,
the query_code=code 1, pages are divided into fifty data pages, and the data from the 215 th page needs to be directly jumped, and the calculation method is as follows:
calculating addup_page_num=215×50= 10750 according to pagesize=50 and pagenum=215; querying the precomputed data store to find a data item of the first eligible piece of add_up_num > = addup_page_num with index of 2. The pieces of data are fetched as es_still and es_etime of 14:22:23:600 and 14:40:58:765, respectively
To obtain data on page 215 using the from+size approach, from=50 (215-1) +1=10701; size=50, call pseudocode as follows:
the hits obtained by the above codes are data specifying a page (target page number), and may be returned to the data requester (data caller).
By the method for paging and inquiring the data by the elastic search provided by the embodiment of the application, a user platform for inquiring the data can be developed, and the user platform can provide that the data checked by the user are stored on the elastic search. When the user wants to view the data, the user can select the target data by clicking the corresponding selection buttons through the selection buttons provided on the platform, and the user can jump to the current page to view the corresponding data due to the fact that the data size is large and the data are displayed on the front-end interface in a paging mode by clicking the corresponding page. When the user platform and the elastic search perform data query interaction, the interface to be queried can meet the requirement of page skip and page turning under deep paging. However, according to the previous discussion, the existing scheme cannot be well supported in terms of efficiency and implementation, and by adopting the technical scheme of the application, the service pain point can be well solved, the page-skipping and page-turning function under deep paging is realized, and a user can randomly page and omnidirectionally browse data.
Based on the same inventive concept, the embodiment of the application also provides an apparatus for paging query data by an elastic search, as described in the following embodiment. Because the principle of the device embodiment for solving the problem is similar to that of the method for paging query data by the elastic search, the implementation of the device embodiment can refer to the implementation of the method, and the repetition is omitted.
Fig. 3 is a schematic diagram of an apparatus for paging query data by elastic search, in an embodiment of the present application, as shown in fig. 3, the apparatus may include: a configuration module 31, a data pre-storage module 32, a data request module 33 and a data query module 34.
The configuration module 31 is configured to obtain preset data query conditions, where the data query conditions include: a query field and a query period; the data pre-storing module 32 is configured to start a timing task according to a data query condition, where the timing task is configured to query, in batch, data meeting a query field in each query period from the elastic search, and store the data in a preset database to obtain a plurality of data blocks, where the data amount contained in each data block does not exceed a preset threshold; the data request module 33 is configured to receive a data query request for the specified page, and search a target data block storing data of the specified page from a plurality of data blocks in a preset database according to the data query request for the specified page; and the data query module 34 is used for querying the data of the designated paging from the target data block.
As can be seen from the above, in the device for paging query data of elastic search provided by the embodiment of the present application, the configuration module 31 obtains the preset data query condition including the query field and the query period; starting a timing task through a data pre-storage module 32 according to preset data query conditions, and querying data meeting query fields in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, wherein the data amount contained in each data block does not exceed a preset threshold value; receiving a data query request for the specified page through the data request module 33, and searching a target data block for storing the data of the specified page from a plurality of data blocks of a preset database according to the data query request for the specified page; finally, the data of the designated page is queried from the target data block through the data query module 34.
According to the device for inquiring data in the elastic search page, which is provided by the embodiment of the application, data are inquired from the elastic search in batches and stored in the preset database in a blocking mode, when a data inquiry request for appointed paging is received, the preset database is searched for storing a data block of the appointed paging data, and the appointed paging data are inquired from the data block, so that the embodiment of the application can be suitable for a scene of deep paging and a scene of page skip access data.
It should be noted that, in the device for paging query data of elastic search provided in the embodiment of the present application, the data query request includes: the data request module 33 is further configured to determine a data entry range of the specified page according to the page size and the page number included in the data query request; and traversing each data block in the preset database, and searching the data block for storing the data of the designated paging according to the data entry range of each data block and the data entry range of the designated paging.
In an alternative implementation, in the device for paging query data by using elastic search provided in the embodiment of the present application, the data query module 34 may be further configured to query data of a specified page from the target data block according to the page size and the page number included in the data query request by adopting a from+size paging query mode of elastic search.
In an optional implementation, the device for paging query data by elastic search provided by the embodiment of the application may further include: a data processing module 35, configured to generate one or more field combination relationships according to each query field included in the data query condition; creating identification information of each field combination relation; and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation.
Optionally, based on the foregoing embodiment, in the device for paging query data by elastic search provided in the embodiment of the present application, the data block information of each data block includes any one of the following: index information of each data block, a data entry range stored on each data block, an accumulated data entry range corresponding to each data block, and query time of first data and last data on each data block.
As an alternative embodiment, the data processing module 35 is further configured to obtain an expression of each field combination relationship; calculating a hash value of each field combination relation expression; and taking the hash value of each field combination relation expression as identification information of each field combination relation.
In an optional implementation, the device for paging query data by elastic search provided by the embodiment of the application may further include: the data output module 36 is configured to return data of a specified page according to the data query request.
The embodiment of the application also provides a computer device, which is used for solving the technical problem that the prior method for inquiring data by paging by the elastomer search cannot realize data access by deep paging down-skip pages.
The embodiment of the application also provides a computer readable storage medium, which is used for solving the technical problem that the prior method for inquiring data by the elastic search paging can not realize the access of data by the deep paging.
In summary, the embodiments of the present application provide a method, an apparatus, a computer device, and a computer readable storage medium for paging query data of an elastic search, which utilize the query characteristics of the elastic search to perform segmentation processing on data in the elastic search through pre-calculation, so that a page-skipping and page-turning function of the elastic search data under deep paging can be implemented under a large data volume, and a user can randomly page and omnidirectionally browse the data. By the embodiment of the application, the query efficiency is improved, and the response speed of the elastic search interface is improved.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the application, and is not meant to limit the scope of the application, but to limit the application to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the application are intended to be included within the scope of the application.
Claims (9)
1. A method of elastic search paging query data, comprising:
acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period;
starting a timing task according to the data query condition, wherein the timing task is used for periodically querying data meeting the query field in each query period in batches from an elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data amount contained in each data block does not exceed a preset threshold value;
receiving a data query request for specified paging, and searching a target data block for storing the data of the specified paging from a plurality of data blocks of the preset database according to the data query request for the specified paging;
querying the data of the designated paging from the target data block;
after acquiring the preset data query condition, the method further comprises the following steps:
generating one or more field combination relations according to each query field contained in the data query condition;
creating identification information of each field combination relation;
and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation.
2. The method of claim 1, wherein the data query request comprises: the method comprises the steps of receiving a data query request for the specified page, searching a target data block for storing the data of the specified page from a plurality of data blocks of the preset database according to the data query request for the specified page, wherein the target data block comprises the following steps:
determining the range of the data items of the designated paging according to the paging size and the number of pages contained in the data query request;
traversing each data block in the preset database, and searching the data block storing the data of the designated paging according to the data entry range of each data block and the data entry range of the designated paging.
3. The method of claim 2, wherein querying the specified page of data from the target block of data comprises:
and inquiring the data of the appointed paging from the target data block according to the paging size and the number of pages contained in the data inquiry request by adopting a from+size paging inquiry mode of an elastic search.
4. The method of claim 1, wherein the data block information for each data block comprises any one of: index information of each data block, a data entry range stored on each data block, an accumulated data entry range corresponding to each data block, and query time of first data and last data on each data block.
5. The method of claim 1, wherein creating identification information for each field combination relationship comprises:
acquiring an expression of each field combination relation;
calculating a hash value of each field combination relation expression;
and taking the hash value of each field combination relation expression as identification information of each field combination relation.
6. The method of any of claims 1 to 5, wherein after querying the specified paged data from the target data block, the method further comprises:
and returning the data of the designated paging according to the data query request.
7. An apparatus for paging query data of an elastomer search, comprising:
the configuration module is used for acquiring preset data query conditions, wherein the data query conditions comprise: a query field and a query period;
the data pre-storage module is used for starting a timing task according to the data query condition, wherein the timing task is used for periodically inquiring data meeting the query field in each query period in batches from the elastic search, and storing the data into a preset database to obtain a plurality of data blocks, and the data quantity contained in each data block does not exceed a preset threshold value;
the data request module is used for receiving a data query request for the specified paging, and searching a target data block for storing the data of the specified paging from a plurality of data blocks of the preset database according to the data query request for the specified paging;
the data query module is used for querying the data of the appointed paging from the target data block;
further comprises: the data processing module is used for generating one or more field combination relations according to each query field contained in the data query condition; creating identification information of each field combination relation; and recording the data block information of one or more data blocks corresponding to each field combination relation in each query period according to the identification information of each field combination relation.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of paging query data of any one of claims 1 to 6 when the computer program is executed.
9. A computer-readable storage medium storing a computer program for executing the method of paging query data of the elastic search of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448278.0A CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448278.0A CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111680063A CN111680063A (en) | 2020-09-18 |
| CN111680063B true CN111680063B (en) | 2023-08-18 |
Family
ID=72453224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010448278.0A Active CN111680063B (en) | 2020-05-25 | 2020-05-25 | Method and device for paging query data by elastic search |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680063B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100247B (en) * | 2020-11-12 | 2021-02-02 | 耀方信息技术(上海)有限公司 | Method and system for querying data by using ElasticSearch |
| CN112650779B (en) * | 2021-01-12 | 2022-09-06 | 浪潮云信息技术股份公司 | Cloud auditing method supporting deep page skipping query based on ElasticSearch |
| CN113360503B (en) * | 2021-06-18 | 2023-02-07 | 建信金融科技有限责任公司 | Test data tracking method and device for distributed database |
| CN114661770A (en) * | 2022-04-11 | 2022-06-24 | 平安资产管理有限责任公司 | Data paging query method and device, computer equipment and readable storage medium |
| CN115098805A (en) * | 2022-08-25 | 2022-09-23 | 广州市千钧网络科技有限公司 | Method, device, equipment and storage medium for inquiring paging data information |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126731A (en) * | 2016-07-01 | 2016-11-16 | 百势软件(北京)有限公司 | A kind of method and device obtaining Elasticsearch paged data |
| CN109325047A (en) * | 2018-11-22 | 2019-02-12 | 北京明朝万达科技股份有限公司 | A kind of interactive mode ElasticSearch depth paging query method and apparatus |
| CN109492148A (en) * | 2018-11-22 | 2019-03-19 | 北京明朝万达科技股份有限公司 | ElasticSearch paging query method and apparatus based on Redis |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110555034A (en) * | 2018-03-28 | 2019-12-10 | 武汉斗鱼网络科技有限公司 | Data query paging method, device, server and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018170276A2 (en) * | 2017-03-15 | 2018-09-20 | Fauna, Inc. | Methods and systems for a database |
-
2020
- 2020-05-25 CN CN202010448278.0A patent/CN111680063B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126731A (en) * | 2016-07-01 | 2016-11-16 | 百势软件(北京)有限公司 | A kind of method and device obtaining Elasticsearch paged data |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110555034A (en) * | 2018-03-28 | 2019-12-10 | 武汉斗鱼网络科技有限公司 | Data query paging method, device, server and medium |
| CN109325047A (en) * | 2018-11-22 | 2019-02-12 | 北京明朝万达科技股份有限公司 | A kind of interactive mode ElasticSearch depth paging query method and apparatus |
| CN109492148A (en) * | 2018-11-22 | 2019-03-19 | 北京明朝万达科技股份有限公司 | ElasticSearch paging query method and apparatus based on Redis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111680063A (en) | 2020-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111680063B (en) | Method and device for paging query data by elastic search | |
| CN107103032B (en) | Mass data paging query method for avoiding global sequencing in distributed environment | |
| CN102542052B (en) | Priority hash index | |
| US8983930B2 (en) | Facet group ranking for search results | |
| US11567681B2 (en) | Method and system for synchronizing requests related to key-value storage having different portions | |
| US9904705B2 (en) | Data table performance optimization | |
| CN108021333B (en) | System, device and method for randomly reading and writing data | |
| US20220138203A1 (en) | Method and system for searching a key-value storage | |
| CN101635001B (en) | Method and apparatus for extracting information from a database | |
| CN114138830A (en) | Second-level query method and device for mass data of big data and computer medium | |
| CN103500188A (en) | Method and system for generating data sheet | |
| CN106874332A (en) | Data bank access method and device | |
| CN111125157B (en) | Query data processing method and device, storage medium and processor | |
| JP7213890B2 (en) | Accelerated large-scale similarity computation | |
| CN106446080B (en) | Data query method, query service equipment, client equipment and data system | |
| CN110019210A (en) | Method for writing data and equipment | |
| CN114116806A (en) | Top-k ranking query and library falling method and device | |
| CN111125155B (en) | Access path-based data query method, device, storage medium and processor | |
| CN110909149B (en) | Data filtering method and device | |
| CN115809248B (en) | Data query method and device and storage medium | |
| CN109934610B (en) | Advertisement audience user data processing method and device | |
| CN111143340B (en) | Data processing method and device, server and client | |
| CN116383255A (en) | Aggregation query method, system, equipment and storage medium | |
| CN117891817A (en) | Data paging method and device, storage medium and electronic equipment | |
| CN117421418A (en) | Text searching method and device based on keywords and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |