CN104298692B - A kind of method and system of distributed search - Google Patents
A kind of method and system of distributed search Download PDFInfo
- Publication number
- CN104298692B CN104298692B CN201310306833.6A CN201310306833A CN104298692B CN 104298692 B CN104298692 B CN 104298692B CN 201310306833 A CN201310306833 A CN 201310306833A CN 104298692 B CN104298692 B CN 104298692B
- Authority
- CN
- China
- Prior art keywords
- server layer
- index
- index request
- request
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 239000000284 extract Substances 0.000 claims abstract description 8
- 238000012795 verification Methods 0.000 claims description 17
- 238000010586 diagram Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method of distributed search, including：Message server layer receives and preserved the index the being verified request that index server layer is sent in real time；When real-time search server layer detects that the message server layer receives new index request, the search server layer in real time extracts the index request from the message server layer, according to the index request generation index file and scans for.The invention also discloses a kind of system of distributed search, and search in real time can be provided using the present invention.
Description
Technical Field
The invention relates to the field of database search, in particular to a distributed search method and a distributed search system.
Background
The search systems used by general enterprises are based on databases, but with the continuous increase of data volume and the continuous increase of search requests, database search cannot meet the requirements. There is a need for enterprises to use distributed search based on search techniques, wherein the distributed search comprises: after the distribution server layer receives the index request, the distribution server layer sends the request to the index server layer server; the index server layer server analyzes the received index request to generate index documents, generates an index file directory after the number of the index documents exceeds a preset number, and synchronizes the index file directory to a storage server layer; the search server periodically acquires the index file directory from the storage server layer and provides search service according to the index file directory.
However, in the distributed search process, since the index files in the search server need to be acquired from the storage server layer, and the index files in the index server layer server are sent to the storage server layer only after reaching the preset number, the process of reaching the search server from the index server layer through the storage server layer from the index server layer takes time, and generally takes time of at least several minutes. It can be seen that the distributed search in the prior art cannot provide real-time search.
Disclosure of Invention
In view of this, the present invention provides a distributed search method and system, which can provide real-time search.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
the invention provides a distributed searching method, which comprises the following steps:
the message server layer receives and stores the index request which is sent by the index server layer in real time and passes the verification;
when the real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, generates an index file according to the index request and searches.
In the above solution, before the message server layer receives and stores the index request which is sent by the index server layer in real time and passes the verification, the method further includes: the index server layer detects whether an index request sent by the distribution server layer is received in real time, if so, the index server layer verifies the index request, and if the index request passes the verification, the index server layer sends the index request to the message server layer; otherwise, the index server layer continuously detects whether an index request is received in real time.
In the above solution, when verifying the index request, the method further includes: periodically sending the index file to a storage server layer; and the search server layer periodically acquires the index file from the storage server layer and searches according to the index file.
In the above solution, when the real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, and generates an index file according to the index request and performs a search, including:
the real-time search server layer detects whether the message server layer receives a new index request in real time, if so, the index request is extracted from the message server layer, and an index file is generated according to the index request and is searched; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
The invention also provides a system for distributed search, comprising: the system comprises a message server layer, a real-time search server layer and an index server layer; wherein,
the message server layer is used for receiving and storing the index request which is sent by the index server layer in real time and passes the verification, and providing a stored new index request for the real-time search server layer;
the real-time search server layer is used for extracting the index request from the message server layer when detecting that the message server layer receives a new index request, generating an index file according to the index request and searching;
and the index server layer is used for sending the verified index request to the message server layer in real time.
In the above solution, the system further includes: the distribution server layer is used for sending an index request to the index server layer;
correspondingly, the index server layer is also used for detecting whether an index request sent by the distribution server layer is received in real time, if so, the index request is verified, and if the index request passes the verification, the index server layer sends the index request to the message server layer; otherwise, the index server layer continuously detects whether an index request is received in real time.
In the above solution, the system further includes: a storage server layer and a search server layer; wherein,
the storage server layer is used for receiving the index file sent by the index server layer;
the search server layer is used for periodically acquiring the index file from the storage server layer and searching according to the index file;
accordingly, the index server layer periodically sends the index file to the storage server layer.
In the above scheme, the real-time search server layer is specifically configured to detect whether the message server layer receives a new index request in real time, and if so, extract the index request from the message server layer, generate an index file according to the index request, and perform search; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
The distributed searching method and the distributed searching system provided by the invention can receive and store the verified index request sent by the index server layer in real time through the message server layer; and when the real-time search server layer detects that the message server layer receives a new index request, extracting the index request, generating an index file according to the index request and searching. Therefore, the index request received by the index server layer can be sent to the real-time search server layer through the message server in real time, and the real-time search server layer performs search operation, so that the problem of poor real-time performance caused by the fact that index files need to be accumulated and then searched in the prior art is solved.
Drawings
FIG. 1 is a schematic flow chart of a distributed search method according to the present invention;
FIG. 2 is a schematic diagram of the system architecture of the distributed search of the present invention;
FIG. 3 is a schematic flow chart of an embodiment of the present invention;
fig. 4 is a schematic diagram of a search server layer structure according to a second embodiment of the present invention.
Detailed Description
The basic idea of the invention is: the message server layer receives and stores the index request which is sent by the index server layer in real time and passes the verification; when the real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, generates an index file according to the index request and searches.
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The distributed search method of the present invention, as shown in fig. 1, includes the following steps:
step 101: and the message server layer receives and stores the verified index request sent by the index server layer in real time.
Here, before executing step 101, the method further includes: the index server layer detects whether an index request sent by a distribution server layer is received or not in real time, if so, the index server layer verifies the index request, and if the index request passes the verification, the index server layer sends the index request to a message server layer; otherwise, the index server layer continuously detects whether an index request is received in real time.
The index request is information issued by an existing distribution server layer in the distributed search architecture, and is an information type specified in the prior art, which is not described herein again.
While verifying the index request, the method further comprises: the index server layer also periodically sends the index file to the storage server layer according to the prior art, and then an index request is returned; the search server layer periodically obtains the index file from the storage server layer, which is not described herein.
The distribution server layer is an existing layer in a distributed search architecture in the prior art, and the functions of the distribution server layer may include: providing an external unified service interface, and receiving and distributing an index request or a search request; unifying the verification request, and filtering out wrong requests; because the index is distributed horizontally on a plurality of search machines, search requests need to be sent to each group of search servers distributed horizontally, and search results need to be merged and sorted.
Step 102: when the real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, generates an index file according to the index request and searches.
Specifically, a real-time search server layer detects whether a new index request is received by the message server layer in real time, if so, the index request is extracted from the message server layer, and an index file is generated according to the index request and is searched; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
As shown in fig. 2, the present invention provides a system for distributed search, comprising: the system comprises a message server layer, a real-time search server layer and an index server layer; wherein,
the message server layer is used for receiving and storing the index request which is sent by the index server layer in real time and passes the verification, and providing a stored new index request for the real-time search server layer;
the real-time search server layer is used for extracting the index request from the message server layer when detecting that the message server layer receives a new index request, generating an index file according to the index request and searching;
and the index server layer is used for sending the verified index request to the message server layer in real time.
The system further comprises: the distribution server layer is used for sending an index request to the index server layer; correspondingly, the index server layer is specifically configured to detect whether an index request sent by the distribution server layer is received in real time, verify the index request if the index request is received, and send the index request to the message server layer if the index request passes the verification; otherwise, continuously detecting whether the index request is received in real time.
The real-time search server layer is specifically used for detecting whether the message server layer receives a new index request in real time, if so, extracting the index request from the message server layer, generating an index file according to the index request and searching; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
The index server layer is also used for verifying the index request issued by the distribution server layer and sending the index request to the message server layer; and periodically putting the index file to the storage server layer, and then returning the index request.
The system further comprises: a storage server layer and a search server layer; the storage server layer is used for receiving the index file sent by the index server layer; the search server layer is used for periodically acquiring the index file from the storage server layer and searching according to the index file; accordingly, the index server layer periodically sends the index file to the storage server layer.
The first embodiment,
The operation flow of the real-time search server layer is shown in fig. 3, and includes:
step 301: and monitoring the message server layer in real time, and if the message server layer receives a new index request, immediately acquiring the condition that the message server layer has the index request by the real-time search server.
Step 302: and generating an index directory in the memory by using the index request.
Here, the time for generating the index directory is very short (since the index data is stored for only several minutes, the memory index directory can be generated within 1 second).
Step 303: and searching by using the index directory.
Example II,
The architecture of the distributed search provided by the invention can comprise: the system comprises a distribution server layer, an index server layer, a search server layer, a real-time search server layer, a storage server layer and a message server layer; wherein,
the distribution server layer is used for providing an external unified service interface, receiving and distributing the index request to the index server layer, or receiving and distributing the search request to the search server; unifying the verification request, and filtering out wrong requests; because the indexes are horizontally distributed on a plurality of search machines, search requests need to be sent to each group of search servers which are horizontally distributed in the search server layer, and search results need to be merged and sorted;
the index server layer is used for sending the index request sent by the distribution server layer to any one index server inside to verify the index request and sending the index request passing the verification to the message server layer; periodically sending the index file to a storage server, and returning an index request;
a search server layer for providing a synchronous indexing and search service from the storage server;
the real-time search server layer is used for extracting the index request from the message server layer when detecting that the message server layer receives a new index request, generating an index file according to the index request and searching;
and the storage server layer is used for storing the index file as a storage bridge of the index server layer and the search server.
And the message server layer is a message bus and is used for providing an index request for the search server layer in real time.
The search server layer is shown in fig. 4, and includes: one or more search servers; each group of search servers in the horizontal direction corresponds to different index files, so that the index files can be expanded horizontally and infinitely; the same index file exists in each group of search servers in the vertical direction, the same search result is returned, and the concurrent access capability is improved; the unlimited extension in the horizontal and vertical directions ensures the expandability and high concurrent access capability of the system.
And the index server layer is used for immediately creating a new index directory when the size of any index directory reaches the configured upper limit value according to the size of the index directory in the configuration file.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (8)
1. A method of distributed searching, the method comprising:
the message server layer receives and stores the index request which is sent by the index server layer in real time and passes the verification;
when a real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, generates an index file according to the index request and searches the index file;
before the message server layer receives and stores the verified index request sent by the index server layer in real time, the method further comprises:
the index server layer detects whether an index request sent by the distribution server layer is received in real time, if so, the index server layer verifies the index request, and if the index request passes the verification, the index server layer sends the index request to the message server layer.
2. The method of claim 1, wherein before the message server layer receives and stores the verified index request sent by the index server layer in real time, the method further comprises:
and the index server layer detects whether the index request sent by the distribution server layer is received in real time, and if the index request is not received, the index server layer continuously detects whether the index request is received in real time.
3. The method of claim 2, wherein the verifying the index request further comprises:
periodically sending the index file to a storage server layer;
and the search server layer periodically acquires the index file from the storage server layer and searches according to the index file.
4. The method according to claim 1, wherein when the real-time search server layer detects that the message server layer receives a new index request, the real-time search server layer extracts the index request from the message server layer, generates an index file according to the index request, and performs a search, including:
the real-time search server layer detects whether the message server layer receives a new index request in real time, if so, the index request is extracted from the message server layer, and an index file is generated according to the index request and is searched; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
5. A system for distributed searching, the system comprising: the system comprises a message server layer, a real-time search server layer and an index server layer; wherein,
the message server layer is used for receiving and storing the index request which is sent by the index server layer in real time and passes the verification, and providing a stored new index request for the real-time search server layer;
the real-time search server layer is used for extracting the index request from the message server layer when detecting that the message server layer receives a new index request, generating an index file according to the index request and searching;
the index server layer is used for sending the index request passing the verification to the message server layer in real time;
the system further comprises: the distribution server layer is used for sending an index request to the index server layer;
correspondingly, the index server layer is further configured to detect whether an index request sent by the distribution server layer is received in real time, verify the index request if the index request is received, and send the index request to the message server layer if the index request passes the verification.
6. The system of claim 5, further comprising: the distribution server layer is used for sending an index request to the index server layer;
correspondingly, the index server layer is further configured to detect whether an index request sent by the distribution server layer is received in real time, and if not, the index server layer continues to detect whether the index request is received in real time.
7. The system of claim 6, further comprising: a storage server layer and a search server layer; wherein,
the storage server layer is used for receiving the index file sent by the index server layer;
the search server layer is used for periodically acquiring the index file from the storage server layer and searching according to the index file;
accordingly, the index server layer periodically sends the index file to the storage server layer.
8. The system of claim 5,
the real-time search server layer is specifically used for detecting whether the message server layer receives a new index request in real time, if so, extracting the index request from the message server layer, generating an index file according to the index request and searching; otherwise, continuously detecting whether the message server layer receives a new index request in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310306833.6A CN104298692B (en) | 2013-07-19 | 2013-07-19 | A kind of method and system of distributed search |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310306833.6A CN104298692B (en) | 2013-07-19 | 2013-07-19 | A kind of method and system of distributed search |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104298692A CN104298692A (en) | 2015-01-21 |
| CN104298692B true CN104298692B (en) | 2017-11-24 |
Family
ID=52318420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310306833.6A Expired - Fee Related CN104298692B (en) | 2013-07-19 | 2013-07-19 | A kind of method and system of distributed search |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104298692B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484694B (en) * | 2015-08-25 | 2019-09-20 | 杭州华为数字技术有限公司 | Full-text search method and system based on distributed data base |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101677328A (en) * | 2008-09-19 | 2010-03-24 | 中兴通讯股份有限公司 | Content-fragment based multimedia distributing system and content-fragment based multimedia distributing method |
| CN102169507A (en) * | 2011-05-26 | 2011-08-31 | 厦门雅迅网络股份有限公司 | Distributed real-time search engine |
| CN103136286A (en) * | 2011-12-05 | 2013-06-05 | 上海特易信息科技有限公司 | Distributed type searching framework |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8903810B2 (en) * | 2005-12-05 | 2014-12-02 | Collarity, Inc. | Techniques for ranking search results |
-
2013
- 2013-07-19 CN CN201310306833.6A patent/CN104298692B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101677328A (en) * | 2008-09-19 | 2010-03-24 | 中兴通讯股份有限公司 | Content-fragment based multimedia distributing system and content-fragment based multimedia distributing method |
| CN102169507A (en) * | 2011-05-26 | 2011-08-31 | 厦门雅迅网络股份有限公司 | Distributed real-time search engine |
| CN103136286A (en) * | 2011-12-05 | 2013-06-05 | 上海特易信息科技有限公司 | Distributed type searching framework |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104298692A (en) | 2015-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106980699B (en) | Data processing platform and system | |
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| US20160239540A1 (en) | Data Query Method and Apparatus, Server, and System | |
| CN110147455B (en) | Face matching retrieval device and method | |
| CN105701096A (en) | Index generation method, data inquiry method, index generation device, data inquiry device and system | |
| CN104572727A (en) | Data querying method and device | |
| CN101826099B (en) | Method and system for identifying similar documents and determining document diffusance | |
| US10503749B2 (en) | Data processing method, device and system of query server | |
| CN111708794B (en) | Data comparison method and device based on big data platform and computer equipment | |
| CN103034735A (en) | Big data distributed file export method | |
| CN106095911A (en) | Search system and method for data synchronization | |
| CN104065715A (en) | Method for resource sharing on basis of cloud platform and cloud platform | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN106326222A (en) | Data processing method and device | |
| CN109739854A (en) | A kind of date storage method and device | |
| CN108090186A (en) | A kind of electric power data De-weight method on big data platform | |
| CN103455491A (en) | Method and device for classifying search terms | |
| CN104298692B (en) | A kind of method and system of distributed search | |
| CN107220003A (en) | A kind of method for reading data and system | |
| CN103646034A (en) | Web search engine system and search method based content credibility | |
| CN102932239A (en) | Communication method and system in instant communication platform | |
| CN113688148B (en) | Urban rail data query method and device, electronic equipment and readable storage medium | |
| CN110895538A (en) | Data retrieval method, device, storage medium and processor | |
| CN105718485B (en) | A kind of method and device by data inputting database | |
| CN109525649B (en) | Data processing method and device for zookeeper client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20171124 |