CN111597212B - Data retrieval method and device - Google Patents
Data retrieval method and device Download PDFInfo
- Publication number
- CN111597212B CN111597212B CN202010445888.5A CN202010445888A CN111597212B CN 111597212 B CN111597212 B CN 111597212B CN 202010445888 A CN202010445888 A CN 202010445888A CN 111597212 B CN111597212 B CN 111597212B
- Authority
- CN
- China
- Prior art keywords
- request
- surface layer
- search
- retrieval
- data retrieval
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 45
- 239000002344 surface layer Substances 0.000 claims description 139
- 238000013507 mapping Methods 0.000 claims description 88
- 238000004458 analytical method Methods 0.000 claims description 27
- 230000004931 aggregating effect Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 description 23
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 9
- 230000002776 aggregation Effects 0.000 description 7
- 238000004220 aggregation Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 239000010410 layer Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 210000002304 esc Anatomy 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application provides a data retrieval method and device, which relate to the field of data processing and comprise the following steps: receiving a data retrieval request; splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests; respectively carrying out data retrieval on the split requests to obtain a plurality of retrieval results which are in one-to-one correspondence with the split requests; and summarizing the plurality of search results to obtain a data search result corresponding to the data search request. Therefore, by implementing the implementation mode, the retrieval speed and the retrieval efficiency can be improved while a large amount of memory of the retrieval server is prevented from being occupied, and the retrieval performance is further improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data retrieval method and apparatus.
Background
With the development of modern network technology and communication technology, people in daily work face the rapid increase of information resource amount, and people want to extract required data from a large amount of information by means of retrieval, so the rapid development of full-text retrieval technology is also rapidly demanded. The existing data retrieval method generally comprises the steps of carrying out comprehensive retrieval in a full database comprising all original data according to a retrieval request input by a user every time, obtaining a retrieval result, and finally returning the retrieval result to the user. However, in practice, it is found that the full database includes the full amount of raw data, and performing the full search every time it is searched consumes a large amount of memory of the search server, resulting in slow search speed and low efficiency.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data retrieval method and apparatus, which can improve retrieval speed and retrieval efficiency while avoiding occupying a large amount of memory of a retrieval server, thereby improving retrieval performance.
An embodiment of the present application provides a data retrieval method, where the data retrieval method includes:
receiving a data retrieval request;
splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests;
respectively carrying out data retrieval on the split requests to obtain a plurality of retrieval results corresponding to the split requests one by one;
and summarizing the plurality of search results to obtain a data search result corresponding to the data search request.
In the implementation process, the method can obtain the data retrieval request preferentially, and then split the data retrieval request according to the preset request splitting rule to obtain a plurality of splitting requests; after a plurality of split requests are acquired, carrying out data retrieval based on each split request to obtain a plurality of retrieval results, wherein the plurality of retrieval results are in one-to-one correspondence with the plurality of split requests; and finally, summarizing a plurality of search results to obtain a final data search result. Therefore, according to the implementation mode, a large amount of data retrieval requests can be split into multiple sub-retrieval through the splitting process, so that a large amount of memory of a retrieval server is prevented from being occupied, meanwhile, the implementation mode can also be used for greatly improving the retrieval efficiency through a multithreading processing mode, in addition, the implementation mode can be simplified, corresponding retrieval is triggered from each detail, and therefore the effect of improving the retrieval performance is achieved.
Further, the step of performing data retrieval on the plurality of split requests to obtain a plurality of retrieval results corresponding to the plurality of split requests one to one includes:
classifying the split requests according to a preset request classification rule to obtain a full-quantity search request set and a surface layer search request set;
performing full-quantity retrieval processing in a preset full-quantity database according to the full-quantity retrieval request set to obtain a full-quantity retrieval result;
performing surface layer search processing in a preset surface layer database according to the surface layer search request set to obtain a surface layer search result;
and determining a plurality of search results corresponding to the plurality of splitting requests one by one according to the full search result and the surface layer search result.
In the implementation process, the method can classify the plurality of split requests preferentially according to a preset request classification rule in the process of acquiring the plurality of search results to obtain two types of search requests, wherein the two types of search requests comprise a search request set formed by a full-quantity search request and a search request set formed by a surface layer search request; after the total search request set and the surface layer search condition set, the method carries out total search processing in a preset total database according to the total search request set to obtain a total search result, and carries out surface layer search processing in the preset surface layer database according to the surface layer search request set to obtain a surface layer search result; and after the full-scale search result and the surface layer search result are obtained, determining a plurality of search results according to the full-scale search result and the surface layer search result. Therefore, by implementing the embodiment, the split request can be classified, so that the search can be divided into two different searches, thereby improving the pertinence of the search and further improving the search precision and the search effect of the search.
Further, the step of performing the surface layer search processing in a preset surface layer database according to the surface layer search request set to obtain a surface layer search result includes:
determining a mapping retrieval request set corresponding to the surface layer retrieval request set in a preset request mapping library;
determining a last update time set formed by the last update time corresponding to each split request in the surface layer search request set according to the mapping search request set;
and performing surface layer search processing in a preset surface layer database according to the last updating time set and the surface layer search request set to obtain a surface layer search result.
In the implementation process, the method can preferentially determine a mapping search request set corresponding to the surface search request set in a preset request mapping library in the process of performing surface search processing to obtain a surface search result, so that the mapping search request set can be used for determining a final update time set formed by the final update time corresponding to each split request in the surface search request set; and then, carrying out surface layer search processing in a preset surface layer database according to the last update time set and the surface layer search request set to obtain a surface layer search result. Therefore, by implementing the implementation mode, the problem of full-text retrieval can be solved by mapping the retrieval request set, the effect of reducing the retrieval pressure is realized, and meanwhile, the error problem caused by data change can be effectively solved by introducing the concept of data update time, so that the retrieval accuracy is improved.
Further, the step of performing surface layer search processing in a preset surface layer database according to the last updated time set and the surface layer search request set to obtain a surface layer search result includes:
acquiring the last database updating time of a preset surface database;
splitting the surface layer search request set according to the last update time set to obtain a first request subset and a second request subset; the last update time corresponding to each split request in the first request subset is smaller than the database update time, and the last update time corresponding to each split request in the second request subset is larger than or equal to the database update time;
determining a mapping retrieval request subset corresponding to the second request subset from the mapping retrieval request set;
performing surface layer search processing in the surface layer database according to the mapping search request subset to obtain a second search result;
and aggregating the second search result to obtain a surface layer search result.
In the implementation process, the method can acquire the update time of the database, and then determine stable data, namely data included in the second request subset, according to the update time of the database and the update time of the search request, so that the partial data can be subjected to surface layer search processing to obtain an accurate second search result. Therefore, by implementing the implementation mode, the method can judge the stability of the data through the update time of the database and the update time of the search request, then the search type is determined according to the stability, and further the corresponding efficient and accurate search is performed after the search type is determined, so that an accurate surface layer search result is obtained.
Further, after the step of splitting the surface layer search request set according to the last updated time set to obtain a first request subset and a second request subset, the method further includes:
performing full-quantity search processing in the full-quantity database according to the first request subset to obtain a first search result;
the step of aggregating the second search results to obtain a surface layer search result comprises the following steps:
and aggregating the first search result and the second search result to obtain a surface layer search result.
In the implementation process, the method can also obtain a first request subset with more active data state, then perform full-scale search processing according to the first request subset to obtain a first search result, and then aggregate the first search result and the second search result to obtain a complete surface layer search result. It can be seen that by implementing this embodiment, the secondary search can be completed in the surface layer search process, and the search accuracy can be improved.
Further, after the receiving the data retrieval request, the method further comprises:
storing the data retrieval request;
acquiring the storage time for storing the data retrieval request;
Performing hot spot query analysis processing on the data retrieval request to obtain an analysis result;
and updating a preset request mapping library according to the storage time and the analysis result.
In the implementation process, the method can store the data retrieval request after the data retrieval request, acquire the storage time of the data retrieval request, and then further perform hot spot query analysis on the data retrieval request to obtain an analysis result, so that the request mapping library can be updated according to the storage time of the data retrieval request and the hot spot analysis result. Therefore, by implementing the embodiment, the updating of the request mapping library can be promoted through the input of the data retrieval request, so that the instantaneity of the request mapping library is realized, and the overall data retrieval effect is improved.
Further, the updating the preset request mapping library according to the storage time and the analysis result includes:
judging whether a target mapping request matched with the data retrieval request is stored in a preset request mapping library according to the analysis result;
if yes, acquiring the target mapping request and the target last update time corresponding to the target mapping request;
Acquiring a target identifier matched with the data retrieval request according to the storage time and the target last update time;
and updating the request mapping library according to the target identifier, the data retrieval request and the last update time of the target.
In the implementation process, the method can detect the target mapping request in advance, acquire the last update time of the target mapping request when the target mapping request is detected, acquire the target identifier matched with the data retrieval request according to the last update time and the storage time of the data retrieval request, and update the request mapping library according to the target identifier, the data retrieval request and the last following time. Therefore, the implementation of the embodiment can accurately finish data updating in real time, thereby ensuring the accuracy of data retrieval.
A second aspect of the embodiments of the present application provides a data retrieval device, including:
a receiving unit for receiving a data retrieval request;
the splitting unit is used for splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests;
The searching unit is used for respectively carrying out data searching on the plurality of split requests to obtain a plurality of searching results corresponding to the plurality of split requests one by one;
and the summarizing unit is used for summarizing the plurality of search results to obtain a data search result corresponding to the data search request.
In the implementation process, the data retrieval device completes the operations of acquiring the data retrieval request, splitting the data retrieval request, retrieving the split requests one by one, and uniformly summarizing the retrieval results through a plurality of units, thereby realizing the function of the device for acquiring and summarizing integration. Therefore, by implementing the implementation mode, the data retrieval device can realize the function of automatic retrieval, so that the data retrieval efficiency is improved, and the data retrieval precision is ensured.
A third aspect of the embodiments of the present application provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to perform the data retrieval method according to any one of the first aspect of the embodiments of the present application.
A fourth aspect of the embodiments of the present application provides a computer readable storage medium storing computer program instructions which, when read and executed by a processor, perform the data retrieval method according to any one of the first aspect of the embodiments of the present application.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a data retrieval method according to an embodiment of the present application;
FIG. 2 is a flowchart of another data retrieval method according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a data retrieval device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of another data retrieval device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Example 1
Referring to fig. 1, fig. 1 is a flowchart of a data retrieval method according to an embodiment of the present application. The method can be applied to various search application scenes, and particularly can be applied to search scenes using an elastic search (full text search engine). The data retrieval method comprises the following steps:
s101, receiving a data retrieval request.
In this embodiment, the data retrieval request may be a word segmentation search class query aggregation request.
In this embodiment, the data retrieval request may be DSL (Domain Specific Language ); the DSL is a JSON format query language supported by the elastic search and generally comprises a query filtering part, an aggregation statistics part, a highlighting part and other auxiliary parts. The basic flow of the query is that the query filters part hit data, and the aggregation statistics part calculates the result according to the hit data.
S102, splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests.
In this embodiment, the preset request splitting rule may be a monthly splitting rule, and the data retrieval request may be split into multiple splitting requests by the request splitting rule.
And S103, respectively carrying out data retrieval on the split requests to obtain a plurality of retrieval results corresponding to the split requests one by one.
In this embodiment, the search results are in one-to-one correspondence with the split requests, that is, one split request corresponds to one search result.
S104, summarizing the plurality of search results to obtain a data search result corresponding to the data search request.
In this embodiment, a plurality of search results are summarized together to obtain a comprehensive data search result.
In the present embodiment, a plurality of search results are all processed in parallel, and therefore, the data search efficiency is made higher.
In this embodiment, although the basic hardware overhead cannot be avoided, the key difference between the method and the traditional scheme is that only data of a non-Text datatype is stored, and therefore, compared with other schemes, the method can greatly reduce the hardware cost, so that the overall hardware consumption only accounts for 5% of the original ESC-M.
In this embodiment, ESC (Elasticsearch Cluster) refers to a cluster service composed of different elastic search nodes, supporting full Text search and big data storage, where a full database stores the ESCs of full raw data, and a surface database does not store Filed of Text datatype compared with the full database. For example, assuming that the data retrieval method described in the present implementation is applied to paper retrieval, the full-volume database stores data of all papers, including paper surface data (including paper topic, paper author identification, paper classification identification, paper comment data, etc.) and paper text data; the surface database stores only the paper surface data, compared to the full database.
In this embodiment, the data retrieval method described in this embodiment is based on an elastic search (full text search engine). The elastosearch is a distributed, highly extended, high real-time search and data analysis engine. The method can conveniently enable a large amount of data to have the capabilities of searching, analyzing and exploring.
In this embodiment, the data retrieval request may be DSL (Domain Specific Language), and DSL is a JSON format query language supported by elastic search, and generally includes a query filtering portion, an aggregate statistics portion, and other auxiliary portions such as highlighting. The basic flow of the query is that the query filters part hit data, and the aggregation statistics part calculates the result according to the hit data.
In this embodiment, the preset request splitting rule includes splitting the data retrieval request according to a preset time threshold.
In this embodiment, the basic unit of the elastic search storage data is documents, each Document is composed of different fields, and one Document must correspond to a unique Field and Index, that is, two documents do not have the same identifier in the same Index. Acquiring a target identifier that matches the data retrieval request, then a unique Document, index and Field can be determined based on the target identifier.
In this embodiment, the execution subject may be a computer, a server, or other computer device having a capability of calculating a search, which is not limited in this embodiment.
In this embodiment, the execution body may be an intelligent electronic device such as a mobile phone or a tablet, which is not limited in this embodiment.
It can be seen that, by implementing the data retrieval method described in fig. 1, the data retrieval request can be preferentially acquired, and then the data retrieval request is split according to a preset request splitting rule, so as to obtain a plurality of split requests; after a plurality of split requests are acquired, carrying out data retrieval based on each split request to obtain a plurality of retrieval results, wherein the plurality of retrieval results are in one-to-one correspondence with the plurality of split requests; and finally, summarizing a plurality of search results to obtain a final data search result. Therefore, according to the implementation mode, a large amount of data retrieval requests can be split into multiple sub-retrieval through the splitting process, so that a large amount of memory of a retrieval server is prevented from being occupied, meanwhile, the implementation mode can also be used for greatly improving the retrieval efficiency through a multithreading processing mode, in addition, the implementation mode can be simplified, corresponding retrieval is triggered from each detail, and therefore the effect of improving the retrieval performance is achieved.
Example 2
Referring to fig. 2, fig. 2 is a flow chart of another data retrieval method according to an embodiment of the present application. The flow chart of the data retrieval method depicted in fig. 2 is modified from the flow chart of the data retrieval method depicted in fig. 1. The data retrieval method comprises the following steps:
s201, receiving a data retrieval request.
In this embodiment, the data retrieval request may be a word segmentation search class query aggregation request.
In this embodiment, the data retrieval request may be DSL (Domain Specific Language ); the DSL is a JSON format query language supported by the elastic search and generally comprises a query filtering part, an aggregation statistics part, a highlighting part and other auxiliary parts. The basic flow of the query is that the query filters part hit data, and the aggregation statistics part calculates the result according to the hit data. An elastosearch is a Lucene-based search server that provides a distributed multi-user-capable full-text search engine.
As an alternative embodiment, after receiving the data retrieval request, the method may further include:
storing the data retrieval request;
acquiring the storage time of a storage data retrieval request;
Performing hot spot query analysis processing on the data retrieval request to obtain an analysis result;
and updating a preset request mapping library according to the storage time and the analysis result.
By implementing the embodiment, the update of the request mapping library can be promoted through the input of the data retrieval request, so that the instantaneity of the request mapping library is realized, and the overall data retrieval effect is improved.
In this embodiment, the data retrieval request may be a search query fusion request.
In this embodiment, the storage time of the data search request, that is, the storage time at which the data search request is stored to start calculation.
In this embodiment, the step of performing the hotspot query analysis processing on the data retrieval request to obtain the analysis result may be understood as analyzing a new hotspot query/updating the existing hotspot query map at regular time, and splitting the request according to a preset rule.
As an alternative implementation mode, when analysis results are obtained, asynchronous execution is adopted to analyze the data search request, the data search request is analyzed for hot spot query sentences, the peak time of full-volume search is avoided, the mapping of the surface layer search to the full-volume search is established, the data search request of the same hot spot is received again, the mapping request corresponding to the data search request can be directly obtained, the surface layer search is carried out according to the mapping request, cluster performance pressure caused by keyword search in the full-volume search is greatly reduced, meanwhile, word segmentation is not dependent in the surface layer search, and compared with the full-volume search, the response can be more timely and rapid.
As a further alternative embodiment, the step of updating the preset request mapping library according to the storage time and the analysis result may include:
judging whether a target mapping request matched with the data retrieval request is stored in a preset request mapping library according to the analysis result;
if yes, acquiring a target mapping request and a target last update time corresponding to the target mapping request;
acquiring a target identifier matched with the data retrieval request according to the storage time and the last update time of the target;
and updating the request mapping library according to the target identification, the data retrieval request and the last updating time of the target.
By implementing the implementation mode, the data updating can be accurately completed in real time, so that the accuracy degree of data retrieval is ensured.
In this embodiment, the step of determining whether the target mapping request matched with the data retrieval request is stored in the preset request mapping library according to the analysis result may be understood as whether the conversion mapping relationship established earlier for the current month already exists.
In this embodiment, if there is a conversion mapping relation established earlier, it is indicated that there is data in the database, and update is required to be deleted; otherwise, the method does not need to be deleted and can be directly updated.
S202, splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests.
In this embodiment, the preset request splitting rule may be a monthly splitting rule, and the data retrieval request may be split into multiple splitting requests by the request splitting rule.
S203, classifying the split requests according to a preset request classifying rule to obtain a full-quantity search request set and a surface layer search request set.
In the present embodiment, the full-volume search request set is a set of a plurality of full-volume search requests for performing full-text search (i.e., full-data search).
In this embodiment, the set of skin search requests is a set of a plurality of skin search requests, where the skin search requests are used for performing non-text search.
S204, performing full-quantity search processing in a preset full-quantity database according to the full-quantity search request set to obtain a full-quantity search result.
In this embodiment, the full database refers to a database having all data.
S205, determining a mapping search request set corresponding to the surface layer search request set in a preset request mapping library.
In this embodiment, the request mapping library refers to a search request library independent of word segmentation, and the search request library corresponds to a large number of table-layer search requests.
In this embodiment, the mapping request in the request mapping library may directly retrieve the history query result corresponding to the surface layer search request, without searching again, so as to achieve the effects of saving time and improving efficiency.
S206, determining a last update time set formed by the last update time corresponding to each split request in the surface layer search request set according to the mapping search request set.
In this embodiment, the last update time in the last update time set corresponds to each split request one by one.
S207, performing surface layer search processing in a preset surface layer database according to the last update time set and the surface layer search request set to obtain a surface layer search result.
In this embodiment, the last update time set is used to obtain the latest retrieval data, so as to improve the accuracy of retrieval.
As an optional implementation manner, the step of performing the surface layer search processing in the preset surface layer database according to the last update time set and the surface layer search request set to obtain the surface layer search result may include:
acquiring the last database updating time of a preset surface database;
splitting the surface layer retrieval request set according to the last update time set to obtain a first request subset and a second request subset; the last update time corresponding to each split request in the first request subset is smaller than the database update time, and the last update time corresponding to each split request in the second request subset is larger than or equal to the database update time;
Determining a mapping retrieval request subset corresponding to the second request subset from the mapping retrieval request set;
performing surface layer search processing in a surface layer database according to the mapping search request subset to obtain a second search result;
and aggregating the second search result to obtain a surface layer search result.
By implementing the embodiment, the stability of the data can be judged by the update time of the database and the update time of the search request, then the search type is determined according to the stability, and further the corresponding efficient and accurate search is performed after the search type is determined, so that an accurate surface layer search result is obtained.
As an optional implementation manner, after the step of splitting the layer search request set according to the last update time set to obtain the first request subset and the second request subset, the method further includes:
performing full-quantity retrieval processing in a full-quantity database according to the first request subset to obtain a first retrieval result;
the step of aggregating the second search result to obtain a surface layer search result comprises the following steps:
and aggregating the first search result and the second search result to obtain a surface layer search result.
By implementing the embodiment, the secondary search can be completed in the surface layer search process, and the search precision is improved
Steps S205 to S207 may constitute steps of performing a skin search process in a preset skin database according to the set of skin search requests to obtain a skin search result. Therefore, by implementing the implementation mode, the problem of full-text retrieval can be solved by mapping the retrieval request set, the effect of reducing the retrieval pressure is realized, and meanwhile, the error problem caused by data change can be effectively solved by introducing the concept of data update time, so that the retrieval accuracy is improved.
S208, determining a plurality of search results corresponding to the plurality of splitting requests one by one according to the full-scale search result and the surface layer search result.
In this embodiment, the plurality of search results are included in either the full-scale search result or the surface-layer search result, and a detailed description is omitted in this embodiment.
S209, summarizing the plurality of search results to obtain a data search result corresponding to the data search request.
As an alternative implementation manner, the updating process is performed on the request mapping library according to the target identifier, the data retrieval request and the target last updating time, and the method further comprises the following steps:
deleting mapping data corresponding to the target identifier in the request mapping library;
Constructing a mapping relation between a target identifier and the data retrieval request;
and adding the mapping relation into the request mapping library to update the request mapping library.
It can be seen that, by implementing the data retrieval method described in fig. 2, the data retrieval request can be preferentially acquired, and then the data retrieval request is split according to a preset request splitting rule, so as to obtain a plurality of split requests; after a plurality of split requests are acquired, carrying out data retrieval based on each split request to obtain a plurality of retrieval results, wherein the plurality of retrieval results are in one-to-one correspondence with the plurality of split requests; and finally, summarizing a plurality of search results to obtain a final data search result. Therefore, according to the implementation mode, a large amount of data retrieval requests can be split into multiple sub-retrieval through the splitting process, so that a large amount of memory of a retrieval server is prevented from being occupied, meanwhile, the implementation mode can also be used for greatly improving the retrieval efficiency through a multithreading processing mode, in addition, the implementation mode can be simplified, corresponding retrieval is triggered from each detail, and therefore the effect of improving the retrieval performance is achieved.
Example 3
Referring to fig. 3, fig. 3 is a schematic structural diagram of a data retrieval device according to an embodiment of the present application. Wherein, this data retrieval device includes:
a receiving unit 310 for receiving a data retrieval request.
And the splitting unit 320 is configured to split the data retrieval request according to a preset request splitting rule, so as to obtain a plurality of split requests.
The retrieving unit 330 is configured to perform data retrieval on the plurality of split requests, and obtain a plurality of retrieval results corresponding to the plurality of split requests one by one.
The summarizing unit 340 is configured to summarize the plurality of search results to obtain a data search result corresponding to the data search request.
In this embodiment, the explanation of the data retrieval device may refer to the description in embodiment 1 or embodiment 2, and the description is not repeated in this embodiment.
Therefore, the data retrieval device described in fig. 3 can complete the operations of obtaining the data retrieval request, splitting the data retrieval request, retrieving the split requests one by one, and collecting the retrieval results uniformly through a plurality of units, thereby realizing the function of the device for integrating obtaining and collecting. Therefore, by implementing the implementation mode, the data retrieval device can realize the function of automatic retrieval, so that the data retrieval efficiency is improved, and the data retrieval precision is ensured.
Example 4
Referring to fig. 4, fig. 4 is a schematic structural diagram of another data retrieval device according to an embodiment of the present application. The schematic structure of the data retrieval device depicted in fig. 4 is modified from the schematic structure of the data retrieval device depicted in fig. 3. Wherein, the search unit 330 includes:
the classifying subunit 331 is configured to classify the plurality of split requests according to a preset request classification rule, so as to obtain a full-volume search request set and a surface layer search request set.
And the retrieval subunit 332 is configured to perform a full-scale retrieval process in a preset full-scale database according to the full-scale retrieval request set, so as to obtain a full-scale retrieval result.
The retrieving subunit 332 is further configured to perform a surface layer retrieving process in a preset surface layer database according to the surface layer retrieving request set, so as to obtain a surface layer retrieving result.
A determining subunit 333, configured to determine a plurality of search results corresponding to the plurality of split requests one-to-one according to the full-scale search result and the surface layer search result.
As an optional implementation manner, when performing the surface layer search processing in a preset surface layer database according to the surface layer search request set to obtain the surface layer search result, the search subunit 332 specifically performs determining a mapping search request set corresponding to the surface layer search request set in a preset request mapping library;
Determining a last update time set formed by the last update time corresponding to each split request in the surface layer search request set according to the mapping search request set;
and performing surface layer search processing in a preset surface layer database according to the final update time set and the surface layer search request set to obtain the surface layer search result.
As an alternative embodiment, the retrieving subunit 332 specifically performs, when performing the operation of performing the skin search processing in the preset skin database according to the last update time set and the skin search request set to obtain the skin search result, obtaining the last database update time of the preset skin database;
splitting the surface layer retrieval request set according to the last update time set to obtain a first request subset and a second request subset; the last update time corresponding to each split request in the first request subset is smaller than the database update time, and the last update time corresponding to each split request in the second request subset is larger than or equal to the database update time;
performing full-quantity retrieval processing in a full-quantity database according to the first request subset to obtain a first retrieval result;
Determining a mapping retrieval request subset corresponding to the second request subset from the mapping retrieval request set;
performing surface layer search processing in a surface layer database according to the mapping search request subset to obtain a second search result;
and aggregating the first search result and the second search result to obtain the operation of the surface layer search result.
As an alternative embodiment, the data retrieval device may further include:
a storage unit 350 for storing the data retrieval request.
An acquisition unit 360 for acquiring a storage time of the stored data retrieval request.
And the analysis unit 370 is configured to perform a hotspot query analysis process on the data retrieval request, so as to obtain an analysis result.
And the updating unit 380 is configured to update the preset request mapping library according to the storage time and the analysis result.
As an alternative embodiment, the updating unit 380 includes:
the judging subunit 381 is configured to judge whether a target mapping request matched with the data retrieval request is stored in the preset request mapping library according to the analysis result.
The obtaining subunit 382 is configured to obtain the target mapping request and the target last update time corresponding to the target mapping request when the determination result of the determining subunit 381 is yes.
The obtaining subunit 382 is further configured to obtain, according to the storage time and the last update time of the target, a target identifier that matches the data retrieval request.
And the updating subunit 383 is configured to update the request mapping library according to the target identifier, the data retrieval request and the target last update time.
In this embodiment, the explanation of the data retrieval device may refer to the description in embodiment 1 or embodiment 2, and the description is not repeated in this embodiment.
Therefore, the data retrieval device described in fig. 4 can complete the operations of obtaining the data retrieval request, splitting the data retrieval request, retrieving the split requests one by one, and collecting the retrieval results uniformly through a plurality of units, thereby realizing the function of the device for integrating obtaining and collecting. Therefore, by implementing the implementation mode, the data retrieval device can realize the function of automatic retrieval, so that the data retrieval efficiency is improved, and the data retrieval precision is ensured.
An embodiment of the present application provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to execute any one of the data retrieval methods of embodiment 1 or embodiment 2 of the present application.
The present embodiments provide a computer readable storage medium storing computer program instructions that, when read and executed by a processor, perform the data retrieval method of any one of embodiment 1 or embodiment 2 of the present application.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (7)
1. A data retrieval method, the data retrieval method comprising:
receiving a data retrieval request;
splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests;
respectively carrying out data retrieval on the split requests to obtain a plurality of retrieval results corresponding to the split requests one by one;
summarizing the plurality of search results to obtain a data search result corresponding to the data search request;
the step of respectively performing data retrieval on the plurality of split requests to obtain a plurality of retrieval results corresponding to the plurality of split requests one by one comprises the following steps:
classifying the split requests according to a preset request classification rule to obtain a full-quantity search request set and a surface layer search request set;
performing full-quantity retrieval processing in a preset full-quantity database according to the full-quantity retrieval request set to obtain a full-quantity retrieval result;
performing surface layer search processing in a preset surface layer database according to the surface layer search request set to obtain a surface layer search result;
determining a plurality of search results corresponding to the plurality of splitting requests one by one according to the full search result and the surface layer search result;
Wherein, the step of performing surface layer search processing in a preset surface layer database according to the surface layer search request set to obtain a surface layer search result comprises the following steps:
determining a mapping retrieval request set corresponding to the surface layer retrieval request set in a preset request mapping library;
determining a last update time set formed by the last update time corresponding to each split request in the surface layer search request set according to the mapping search request set;
performing surface layer search processing in a preset surface layer database according to the last update time set and the surface layer search request set to obtain a surface layer search result;
wherein, the step of performing surface layer search processing in a preset surface layer database according to the last updated time set and the surface layer search request set to obtain a surface layer search result comprises the following steps:
acquiring the last database updating time of a preset surface database;
splitting the surface layer search request set according to the last update time set to obtain a first request subset and a second request subset; the last update time corresponding to each split request in the first request subset is smaller than the database update time, and the last update time corresponding to each split request in the second request subset is larger than or equal to the database update time;
Determining a mapping retrieval request subset corresponding to the second request subset from the mapping retrieval request set;
performing surface layer search processing in the surface layer database according to the mapping search request subset to obtain a second search result;
and aggregating the second search result to obtain a surface layer search result.
2. The data retrieval method according to claim 1, wherein after the step of splitting the set of skin retrieval requests according to the last updated time set to obtain a first subset of requests and a second subset of requests, the method further comprises:
performing full-quantity search processing in the full-quantity database according to the first request subset to obtain a first search result;
the step of aggregating the second search results to obtain a surface layer search result comprises the following steps:
and aggregating the first search result and the second search result to obtain a surface layer search result.
3. The data retrieval method according to claim 1, wherein after the receiving of the data retrieval request, the method further comprises:
storing the data retrieval request;
acquiring the storage time for storing the data retrieval request;
Performing hot spot query analysis processing on the data retrieval request to obtain an analysis result;
and updating a preset request mapping library according to the storage time and the analysis result.
4. The data retrieval method according to claim 3, wherein the updating the preset request mapping library according to the storage time and the analysis result comprises:
judging whether a target mapping request matched with the data retrieval request is stored in a preset request mapping library according to the analysis result;
if yes, acquiring the target mapping request and the target last update time corresponding to the target mapping request;
acquiring a target identifier matched with the data retrieval request according to the storage time and the target last update time;
and updating the request mapping library according to the target identifier, the data retrieval request and the last update time of the target.
5. A data retrieval device, the data retrieval device comprising:
a receiving unit for receiving a data retrieval request;
the splitting unit is used for splitting the data retrieval request according to a preset request splitting rule to obtain a plurality of splitting requests;
The searching unit is used for respectively carrying out data searching on the plurality of split requests to obtain a plurality of searching results corresponding to the plurality of split requests one by one;
a summarizing unit, configured to summarize the plurality of search results to obtain a data search result corresponding to the data search request;
wherein the retrieval unit includes:
the classifying sub-unit is used for classifying the plurality of split requests according to a preset request classifying rule to obtain a full-quantity search request set and a surface layer search request set;
the searching subunit is used for carrying out full-quantity searching processing in a preset full-quantity database according to the full-quantity searching request set to obtain a full-quantity searching result;
the searching subunit is further configured to perform surface layer searching processing in a preset surface layer database according to the surface layer searching request set, so as to obtain a surface layer searching result;
the determining subunit is used for determining a plurality of search results corresponding to the plurality of splitting requests one by one according to the full search result and the surface layer search result;
the searching subunit performs surface layer searching processing in a preset surface layer database according to the surface layer searching request set, and specifically executes mapping searching request set corresponding to the surface layer searching request set in a preset request mapping library when the operation of obtaining the surface layer searching result is performed;
Determining a last update time set formed by the last update time corresponding to each split request in the surface layer search request set according to the mapping search request set;
performing surface layer search processing in a preset surface layer database according to the last update time set and the surface layer search request set to obtain a surface layer search result;
the searching subunit performs surface layer searching processing in a preset surface layer database according to the last updating time set and the surface layer searching request set, and specifically performs database updating time for obtaining the last time of the preset surface layer database when the operation of obtaining the surface layer searching result is performed;
splitting the surface layer retrieval request set according to the last update time set to obtain a first request subset and a second request subset; the last update time corresponding to each split request in the first request subset is smaller than the database update time, and the last update time corresponding to each split request in the second request subset is larger than or equal to the database update time;
determining a mapping retrieval request subset corresponding to the second request subset from the mapping retrieval request set;
Performing surface layer search processing in a surface layer database according to the mapping search request subset to obtain a second search result;
and aggregating the second search result to obtain a surface layer search result.
6. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the data retrieval method of any one of claims 1 to 4.
7. A readable storage medium, characterized in that the readable storage medium has stored therein computer program instructions, which when read and executed by a processor, perform the data retrieval method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010445888.5A CN111597212B (en) | 2020-05-22 | 2020-05-22 | Data retrieval method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010445888.5A CN111597212B (en) | 2020-05-22 | 2020-05-22 | Data retrieval method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111597212A CN111597212A (en) | 2020-08-28 |
| CN111597212B true CN111597212B (en) | 2024-03-08 |
Family
ID=72186400
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010445888.5A Active CN111597212B (en) | 2020-05-22 | 2020-05-22 | Data retrieval method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111597212B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6678674B1 (en) * | 1998-07-09 | 2004-01-13 | Informex, Inc. | Data retrieving method and apparatus data retrieving system and storage medium |
| CN1987853A (en) * | 2005-12-23 | 2007-06-27 | 北大方正集团有限公司 | Searching method for relational data base and full text searching combination |
| CN107870985A (en) * | 2017-10-12 | 2018-04-03 | 深圳市金立通信设备有限公司 | A kind of method, server and computer-readable recording medium for retrieving information |
-
2020
- 2020-05-22 CN CN202010445888.5A patent/CN111597212B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6678674B1 (en) * | 1998-07-09 | 2004-01-13 | Informex, Inc. | Data retrieving method and apparatus data retrieving system and storage medium |
| CN1987853A (en) * | 2005-12-23 | 2007-06-27 | 北大方正集团有限公司 | Searching method for relational data base and full text searching combination |
| CN107870985A (en) * | 2017-10-12 | 2018-04-03 | 深圳市金立通信设备有限公司 | A kind of method, server and computer-readable recording medium for retrieving information |
Non-Patent Citations (1)
| Title |
|---|
| 陆群 ; .中国网络之门争夺战悄然进行.软件.1998,(09),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111597212A (en) | 2020-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106033416B (en) | Character string processing method and device | |
| CN109947904B (en) | Preference space Skyline query processing method based on Spark environment | |
| KR100544514B1 (en) | Method and system for determining relation between search terms in the internet search system | |
| CN107180093B (en) | Information searching method and device and timeliness query word identification method and device | |
| US20090182723A1 (en) | Ranking search results using author extraction | |
| US20080147642A1 (en) | System for discovering data artifacts in an on-line data object | |
| WO2014056397A1 (en) | Label of interest recommendation method, system and computer readable medium | |
| CN103049575A (en) | Topic-adaptive academic conference searching system | |
| US20140156606A1 (en) | Method and System for Integrating Data Into a Database | |
| CN112115227A (en) | Data query method and device, electronic equipment and storage medium | |
| De Boom et al. | Semantics-driven event clustering in Twitter feeds | |
| CA2906767A1 (en) | Non-deterministic disambiguation and matching of business locale data | |
| CN113220904A (en) | Data processing method, data processing device and electronic equipment | |
| CN110580255A (en) | method and system for storing and retrieving data | |
| CN109656947B (en) | Data query method and device, computer equipment and storage medium | |
| CN112052259A (en) | Data processing method, device, equipment and computer storage medium | |
| CN111597212B (en) | Data retrieval method and device | |
| CN110705297A (en) | Enterprise name-identifying method, system, medium and equipment | |
| US20150154195A1 (en) | Method for entity-driven alerts based on disambiguated features | |
| CN111639099A (en) | Full-text indexing method and system | |
| CN110781309A (en) | Entity parallel relation similarity calculation method based on pattern matching | |
| US20190303464A1 (en) | Directed Data Indexing Based on Conceptual Relevance | |
| Zhang et al. | An approximate approach to frequent itemset mining | |
| CN105224567B (en) | Keyword searching method and device | |
| CN114579573B (en) | Information retrieval method, information retrieval device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: 15-5, 1st Floor, Building 4, No. 15 Haidian Middle Street, Haidian District, Beijing, 100082 Applicant after: Beijing minglue Zhaohui Technology Co.,Ltd. Address before: Room 2020, 2nd floor, building 27, 25 North Third Ring Road West, Haidian District, Beijing Applicant before: BEIJING SUPERTOOL INTERNET TECHNOLOGY LTD. Country or region before: China |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |