CN111625701A - Search method, search device, server and storage medium - Google Patents

Search method, search device, server and storage medium Download PDF

Info

Publication number
CN111625701A
CN111625701A CN202010451012.1A CN202010451012A CN111625701A CN 111625701 A CN111625701 A CN 111625701A CN 202010451012 A CN202010451012 A CN 202010451012A CN 111625701 A CN111625701 A CN 111625701A
Authority
CN
China
Prior art keywords
index
server
retrieval
target
keyword
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010451012.1A
Other languages
Chinese (zh)
Other versions
CN111625701B (en
Inventor
李俊良
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to CN202010451012.1A priority Critical patent/CN111625701B/en
Publication of CN111625701A publication Critical patent/CN111625701A/en
Application granted granted Critical
Publication of CN111625701B publication Critical patent/CN111625701B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/951Indexing; Web crawling techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9538Presentation of query results
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention discloses a searching method, a searching device, a server and a storage medium. The method comprises the steps of determining keywords; searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode; and sorting according to the index result to obtain a retrieval result.

Description

Search method, search device, server and storage medium
Technical Field
The present invention relates to the field of data processing, and in particular, to a search method, apparatus, server, and storage medium.
Background
At present, the search technology is developed vigorously, the search cannot be conducted on calculation and storage in the distributed field, and currently, the index amount reaches a very large scale along with more and more webpage resources.
However, in the related art, due to the limitations of network Input Output (IO) and computing power, more results cannot be retrieved and sorted.
Disclosure of Invention
In order to solve the existing technical problem, embodiments of the present invention provide a search method, apparatus, server, and storage medium.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides a searching method, which is applied to a retrieval server and comprises the following steps:
determining a keyword;
searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on a Remote Direct Memory Access (RDMA) mode;
and sorting according to the index result to obtain a retrieval result.
In the above scheme, the searching for the index result matching with the keyword from the index server based on the keyword includes:
sending a query request to the index server, wherein the query request comprises the keyword; the query request is used for the index server to perform index query according to the key words;
and receiving an index result sent by the index server.
In the above scheme, the method further comprises:
determining a retrieval mode, and determining a first target time length and a second target time length based on the retrieval mode;
the first target duration represents the duration of the index result; and the second target time length representation obtains the time length of the retrieval result based on the index result sorting.
In the foregoing solution, the determining the first target duration and the second target duration based on the search mode includes:
when the retrieval mode is the first mode, determining the first target time length based on the time length required for searching the index result meeting the first quantity requirement from the index server;
determining the second target duration based on a target delay duration and the first target duration; the target delay time represents the time allowed for retrieval.
In the foregoing solution, the determining the first target duration and the second target duration based on the search mode includes:
when the retrieval mode is a second mode, determining the second target time length based on the time length required for sorting according to the index result;
determining the first target duration based on a target delay duration and the second target duration; the target delay time represents the time allowed for retrieval.
In the above scheme, the searching for the index result matching with the keyword from the index server based on the keyword includes:
and searching an index result matched with the keyword from an index server based on the keyword in the first target time length.
In the foregoing scheme, the sorting according to the index result to obtain a retrieval result includes:
and sequencing according to the index result within the second target time length to obtain a retrieval result.
The embodiment of the present invention further provides a search device, which is arranged on a retrieval server, and includes:
an acquisition unit configured to determine a keyword;
the first searching unit is used for searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode;
and the processing unit is used for sequencing according to the index result to obtain a retrieval result.
In the above scheme, the first search unit is configured to send a query request to the index server, where the query request includes the keyword; the query request is used for the index server to perform index query according to the key words; and receiving an index result sent by the index server.
In the above scheme, the apparatus further comprises: the device comprises a first determining unit, a second determining unit and a control unit, wherein the first determining unit is used for determining a retrieval mode and determining a first target time length and a second target time length based on the retrieval mode; the first target duration represents the duration of the index result; and the second target time length representation obtains the time length of the retrieval result based on the index result sorting.
In the foregoing solution, the first determining unit is specifically configured to determine the first target time length based on a time length required to search an index result meeting a first quantity requirement from an index server when the retrieval mode is the first mode; determining the second target duration based on a target delay duration and the first target duration; the target delay time represents the time allowed for retrieval.
In the foregoing solution, the first determining unit is specifically configured to determine the second target duration based on a duration required for sorting according to the index result when the retrieval mode is the second mode; determining the first target duration based on a target delay duration and the second target duration; the target delay time represents the time allowed for retrieval.
In the foregoing scheme, the first searching unit is configured to search, within the first target duration, an index result matched with the keyword from an index server based on the keyword.
In the foregoing scheme, the first searching unit is configured to perform sorting according to the index result within the second target duration to obtain a retrieval result.
An embodiment of the present invention further provides a server, including: a processor and a memory for storing a computer program capable of running on the processor,
wherein the processor is configured to perform the steps of any of the above methods when running the computer program.
An embodiment of the present invention further provides a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of any one of the above methods.
According to the searching method, the searching device, the server and the storage medium provided by the embodiment of the invention, the keyword is determined through the retrieval server; searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode; the retrieval results are obtained by sequencing according to the indexing results, and the retrieval server queries the indexing server by using an RDMA mode, so that the query delay can be reduced; by reducing the query delay, the overall search delay can be reduced, and higher-speed retrieval experience can be provided for the user, or more results can be queried within the allowed delay time, so that more comprehensive retrieval results can be provided for the user; and finally, the search experience of the user can be improved.
Drawings
FIG. 1 is a schematic diagram of a distributed search architecture;
FIG. 2 is a schematic diagram of a search and sort separation
FIG. 3 is a schematic diagram of a TCP/IP based communication;
fig. 4 is a schematic flowchart of a searching method according to an embodiment of the present invention;
fig. 5 is a schematic diagram of data transmission based on an RDMA manner according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an RDMA implementation according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a search architecture based on an RDMA manner according to an embodiment of the present invention;
FIG. 8 is a diagram comparing Omni-Path and Infiniband technologies based on RDMA;
fig. 9 is a schematic structural diagram of a search apparatus according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
It should be noted that: in the present examples, "first", "second", etc. are used for distinguishing similar objects and are not necessarily used for describing a particular order or sequence.
In addition, the technical solutions described in the embodiments of the present invention may be arbitrarily combined without conflict.
In the present examples, a plurality means at least two, e.g., two, three, etc., unless specifically limited otherwise.
As mentioned above, currently, as the web page resources are more and more, the index amount has reached a very large scale, and in order to optimize the index query and sorting performance, the following ways are provided:
the first mode is as follows: the full-network search engines such as Baidu search engines and Shenma search engines construct indexes by dividing documents or word lists of resources, and solve the problems of index query and sequencing performance optimization by utilizing distributed capacity; as the architecture shown in FIG. 1 is adopted, the divided resources are respectively stored in a plurality of nodes (nodes), and the nodes comprise N11……Nnm(ii) a An access statistics server (mergeserver) can acquire data from nodes through a reliable coordination system (such as zookeeper) of a distributed system;
the second mode is as follows: the vertical search engines such as the Ali search engine and the Mei Tuo search engine carry out targeted optimization on fields of the documents on the basis of document or word list division so as to improve the index query and sequencing performance.
With the continuous increase of the index amount, the node needs to use a higher-configuration memory (specifically, more than 1T memory) or separate retrieval (including matching and filtering) and sorting, that is, a server for retrieval and a server for sorting are provided, and the server for sorting acquires data from the server for retrieval and sorts the data according to the acquired data, as shown in fig. 2.
However, the retrieval and the sequencing are separated, and although the problem of single-machine memory limitation is solved, the network loss is also brought, including but not limited to the problem of transmission delay and the problem of computing capacity limitation; specifically, since the conventional network uses a Transmission Control Protocol/Internet Protocol (TCP/IP) for Transmission, operations such as data packaging, moving, and copying all consume the performance of a Central Processing Unit (CPU); in addition, since retrieval and sorting are performed by heavy CPU calculation in the searching process, the acquisition of a large amount of data for sorting is limited by network IO and computing capacity; based on the above problems, more results cannot be retrieved for sorting, which causes a certain problem in the efficiency and accuracy of the current search.
As technology has evolved, auxiliary processors (co-processors) have also been proposed for searching; specifically, a part of light weight calculation is put into a storage service for calculation in an assisted manner, but the network IO bottleneck cannot be solved, and although the distance of sequencing and retrieval is saved by local calculation, the local calculation is only limited to a light weight calculation scene. The co-processor, which may also be referred to herein as a co-processor, is a processor developed and applied to assist a central processing unit in performing tasks that it cannot perform or that it performs inefficiently. HBase distributed storage uses the co-processor.
FIG. 3 is a schematic diagram of a TCP/IP based communication; as shown in fig. 3, the communication of the related Search technology mainly adopts a TCP/IP technology (since a User Datagram Protocol (UDP) technology is unreliable and is less used for Search communication), a User searches data through a Search Application (Search Application), and needs to go from a User mode to a network card, and needs to perform data packing, moving and copying for many times in the middle; for searching data, index and calculation (including calculation related to matching, filtering and the like) are in the same node, the calculation capacity of a CPU is limited by the calculation capacity of the node, especially in a distributed environment, if part of data is sent to another idle machine through a network for calculation, delay cannot be guaranteed, for example, an open-source distributed compiling tool (Distcc) adopts a similar mode, and a low-delay condition does not need to be considered in the scenario of compiling; however, the search request is time-consuming only for a limited time (generally, the search request is required to be returned within 500 ms), and if the time limit is exceeded, the user experience is reduced, so that the separation of the index and the calculation during the search is difficult to realize to ensure the user experience, and therefore, a more complex architecture and the division of complex indexes are required to meet the expansion of the large data index.
In addition, the inverted index can reach hundreds of thousands of documents even in millions of chains in the rough ranking stage, but in the precise ranking stage, if the inverted index can reach hundreds of thousands of documents in a local mode, if the inverted index is in a service mode, only hundreds of documents are involved in the precise ranking in consideration of network influence, and therefore the retrieval result effect is poor.
In summary, the search techniques in the related art all have certain drawbacks.
Based on this, in various embodiments of the invention, keywords are determined by the search server; searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode; and sorting according to the index result to obtain a retrieval result.
Fig. 4 is a schematic flowchart of a searching method according to an embodiment of the present invention; as shown in fig. 4, the method is applied to a retrieval server, and the method includes:
step 401, determining keywords;
step 402, searching an index result matched with the keyword from an index server based on the keyword;
the retrieval server is communicated with the index server based on a Remote Direct Memory Access (RDMA) mode;
and 403, sorting according to the index result to obtain a retrieval result.
In step 401, the determining the keyword includes: the search server determines the input keyword. In actual application, a user can input keywords through a search interface displayed on a display screen of the terminal, the terminal is communicated with the retrieval server, and the terminal determines the keywords and then sends the keywords to the retrieval server; of course, the keyword may also be determined by the search server itself, that is, the search server may be provided with a human-computer interaction interface, and the user may directly operate the search server to input the keyword.
The keywords are used to query data (i.e., searched data) desired by the user. In practical applications, the keywords may be characters of various languages, such as words, pinyin, symbols, or numbers.
In step 402, the keywords are directly used as a search index.
In one embodiment, the searching the index result matched with the keyword from the index server based on the keyword comprises:
sending a query request to the index server, wherein the query request comprises the keyword; the query request is used for the index server to perform index query according to the key words;
and receiving an index result sent by the index server.
The index result comprises at least one datum searched according to the key words.
Here, since the search server communicates with the index server based on the RDMA scheme, the delay time for obtaining the index result is short.
In practical application, in order to obtain more relevant data as an index result, or in order to leave more time for calculation to obtain data with higher relevance to the keyword, and finally determine a search result with higher accuracy, a search mode may be specifically determined, and time for search may be allocated based on the search mode.
Based on this, in an embodiment, the method further comprises:
determining a retrieval mode, and determining a first target time length and a second target time length based on the retrieval mode;
the first target duration represents the duration of the index result;
and the second target time length representation obtains the time length of the retrieval result based on the index result sorting.
In one embodiment, the determining the first target duration and the second target duration based on the retrieval mode includes:
when the retrieval mode is the first mode, determining the first target time length based on the time length required for searching the index result meeting the first quantity requirement from the index server;
determining the second target duration based on a target delay duration and the first target duration; the target delay time represents the time allowed for retrieval.
In this way, after the duration (i.e., the first target duration) required for obtaining a certain number of index results (i.e., index results meeting the first number of requirements) by the query is determined, the duration (i.e., the second target duration) for performing corresponding calculation is determined based on the remaining duration, and more time is left for performing corresponding calculation (including calculation required for sorting based on the index results).
In one embodiment, the determining the first target duration and the second target duration based on the retrieval mode includes:
when the retrieval mode is a second mode, determining the second target time length based on the time length required for sorting according to the index result;
determining the first target duration based on a target delay duration and the second target duration; the target delay time represents the time allowed for retrieval.
In this way, the time length required for sorting (i.e., the second target time length) is determined first, and then the time length for querying (i.e., the first target time length) is determined based on the remaining time length, so that more index results can be obtained within a certain allowed time delay time length.
Specifically, to ensure the user experience, the delay time allowed for one search is typically within 500 milliseconds (ms), and if the delay time is exceeded, the user experience becomes worse.
Assuming that the target delay time can be 500ms, that is, index results need to be searched from an index server within 500ms, and sequencing is performed according to the index results, so as to obtain a retrieval result for feeding back to a user;
in the sorting process, it is assumed that a first time period is consumed for obtaining a first number of index results, and a second time period (assumed to be 10ms) is approximately consumed for fine sorting a second number (assumed to be 200) of index results.
If the total time (namely the target delay time length) is kept unchanged and the number of the index results to be inquired is unchanged (namely the first target time length to be consumed can be determined), the consumption of the first target time length can be reduced in an RDMA mode, so that a longer second target time length can be reserved, the index results can be sorted more accurately by applying more time lengths in the sorting process, and the retrieval results with more accuracy and better relevance can be obtained.
Correspondingly, if the total time consumption (namely the target delay time length) is kept unchanged and the time length to be sorted is unchanged (namely the second target time length to be consumed can be determined), the remaining time length is used as the time length for query, namely more first target time lengths can be left, more index results can be obtained for sorting by obtaining more first target time lengths, and therefore more accurate retrieval results with better relevance can be obtained.
The method of the embodiment of the invention can not only improve the sorting performance, but also is applicable to retrieval results and the like.
In one embodiment, the searching the index result matched with the keyword from the index server based on the keyword comprises:
and searching an index result matched with the keyword from an index server based on the keyword in the first target time length.
In an embodiment, the sorting according to the index result to obtain a retrieval result includes:
and sequencing according to the index result within the second target time length to obtain a retrieval result.
In an embodiment, the method further comprises:
and establishing a communication relationship with the index server based on an RDMA protocol.
Here, after establishing a communication relationship with the index server based on the RDMA protocol, the search server and the index server may perform data transmission based on the RDMA scheme.
Corresponding to the method shown in fig. 4, an embodiment of the present invention further provides a search method applied to an index server, including:
receiving a keyword sent by a retrieval server;
searching an index result matched with the keyword, and sending the index result to a retrieval server; the index result is used for the search server to sort so as to obtain a search result;
here, the search server may communicate with the index server based on an RDMA manner.
Here, the index server may be an index server group, that is, a plurality of parallel index servers are adopted; each index server and the retrieval server can communicate based on RDMA mode; therefore, as the index server is increased, the storage space and the calculation power are increased.
The scheme of the embodiment of the invention not only is simple network protocol replacement through RDMA technology, but also can use a remote memory (namely an index server) as a local memory to a certain extent, so that the calculation and the index of the search are completely separated and layered, and more calculation power is obtained in the sequencing direction of the heavy calculation, so that a user can enjoy low time delay on senses, and the correlation effect of the whole body is more comprehensively improved.
Fig. 5 is a schematic diagram of data transmission based on an RDMA manner according to an embodiment of the present invention; as shown in fig. 5, the user-mode search application may directly access the network card through an RDMA verbs interface (RDMA verbs interface), and compared with the data transmission based on the TCP/IP method shown in fig. 3, the user-mode search application skips the kernel-mode process in fig. 3, thereby achieving the purpose of direct data exchange between two communication nodes and reducing time consumption. In addition, since the kernel mode is not needed, more computing power is left in the CPU for searching, specifically, more computing power is left for searching parts needing computing, such as retrieval (including matching, filtering) and sorting.
Implementations of RDMA include any of the following: InfiniBand (IB), RDMA over Converged Ethernet (RoCE), Internet Wide area RDMA Protocol (iWARP), as shown in FIG. 6; wherein, IB is a computer network communication standard for high-performance computation, which has extremely high throughput and extremely low delay and is used for data interconnection between computers; RoCE is a network protocol that allows remote direct memory access over an Ethernet network; iWARP is also a computer network protocol that preserves the TCP/IP protocol for remote direct memory access. IB realizes RDMA completely from a hardware layer, has high efficiency, but is relatively expensive; RoCE is also based on Ethernet, and comprises two versions of V1 and V2, the V1 version is based on the Media Access Control (MAC) address of the L2 layer, and is mainly applied to local area networks, the V2 version is based on UDP, and the RDMA data packets are packaged for the second time, and the performance is slightly inferior to IB, but the cost is low; iWARP is based on TCP, and the performance is not good than the former two; either way, hardware is relied upon and uniform packaging is done at the RDMA interface layer (RDMA API). In practical application, any mode can be selected according to requirements, and the method is not limited here.
Fig. 7 is a schematic diagram of a search architecture based on an RDMA method according to an embodiment of the present invention; as shown in fig. 7, the search Server (SearchServer) and the Index Server Group (Index Server Group) communicate based on the RDMA scheme.
When the retrieval server queries the index server, the RDMA protocol provides a Message Queue (MQ) based peer-to-peer communication, and each retrieval server can directly acquire data of each node (i.e., the index server) in the index server group without intervention of an operating system and a protocol stack.
The searching method comprises the following steps:
step 701, establishing an RDMA communication mode between a retrieval server and an index server;
the step 701 specifically includes:
1. RDMA device initialization procedure: acquiring the number of devices which can use RDMA transmission and related information of the devices; the RDMA device includes: a search server (SearchServer); the apparatus using RDMA transfer includes: an Index Server group (including a plurality of Index servers);
2. create Queue pair (QP, Queue Pairs) information: during communication, the retrieval server creates a Channel (Channel) connection, and the head and tail end points of each Channel are two pairs of QPs; each pair of QPs consists of a Send Queue (SQ) and a Receive Queue (RQ), and the QPs are mapped to the virtual address space of the application so that the application accesses the direct memory access (RNIC) network card directly through it.
3. Registering network card memory information: registering an operating system physical memory of the RDMA equipment to a network card inserted in the RDMA equipment;
4. exchanging QP information; specifically, after the retrieval server (i.e. a client) creates the QP, the modified state level reset (reset) is reset to initialization (INIT), the initialization is modified to the remote request to send bit (RTR), and the bit is sent to the index server (i.e. a server); the index server creates QP, the modification state machine has INIT to RTR and sends to the retrieval server, the retrieval server modification state machine has RTR to Request to send bit (RTS, Request to send) and sends to the index server, the index server modification state machine has RTR to RTS, and thus RMDA link establishment is completed.
The retrieval server and the index server can communicate in an RDMA mode through the steps.
Here, the index server may be an index server group including a plurality of index servers, each of which performs parallel processing (MapReduce). The retrieval server communicates with each node (i.e., index server) based on the RMDA method.
Step 702, the search server and the index server perform data transmission.
The step 702 specifically includes:
sending a query request to the index server, wherein the query request comprises the keyword; the query request is used for the index server to perform index query according to the key words; and receiving an index result sent by the index server.
After the data transmission is finished, the method further comprises the following steps: the Channel connection is closed.
The following further illustrates the search for RDMA approach.
In a user create transfer Request Work application (WR), which describes message data that a retrieval server wishes to transfer to the opposite end of the Channel (i.e., the index server), the WR is dropped to a certain queue Work queue (WQ, WorkQueue) in the QP. In WQ, WRs created by users are converted into the format of Work Queue Elements (WQEs), asynchronous scheduling analysis of RNIC is waited, and real messages are taken from a Buffer register (Buffer) pointed by the WQEs and sent to a Channel opposite end, namely an index service end. The RDMA protocol may provide a message queue Complete Queue (CQ) that is used to know that a message on a retrieve server WQ has been processed.
Based on RDMA mode search, the index server provides an interface corresponding to index memory query, memory data can be Zero-copied (Zero Copy, which means no memory Copy, reduces system swap and call out, and improves efficiency), a kernel is bypassed (KernelBypass, which means that the function of a kernel subsystem is not used, codes with the same function realized by the index server are adopted to process or directly access and control the device memory from a user space to avoid data from being copied from the device to the kernel and then from the kernel to the user space), and efficient access is directly obtained, and the effect is shown in table 1:
TCP/IP RDMA
speed of transmission 10Gbps 100Gbps
Time consuming 20us 1us
TABLE 1 TCP/IP vs RDMA transport Performance comparison
Assuming that the time required for the sorting process is about 10ms, which is generally 200 units of fine sorting, if the total time consumption is maintained unchanged, the document data can be greatly increased to more than 4000 by using the RDMA method.
It should be noted that, for the single-side operation (e.g. RDMA WRITE), only the virtual address directly accessing the remote is provided, and the participation of the remote application is not needed, so that the method is suitable for bulk data transmission; for bilateral operation (such as RDMA SEND/RECEIVE), similar to a bottom-layer database Buffer Pool (Buffer Pool) of a traditional network, the participation processes of a transmitting party and a receiving party are not different, and are different in zero copy and Kernel Bypass.
Since hardware-based implementation of id (infiniband) is best performing but expensive, IBM proposes an Omni-Path technology replacement, but is still the RDMA protocol. FIG. 8 is a schematic diagram comparing Omni-Path and Infiniband technologies based on RDMA, as shown in FIG. 8, the single-hop delay reaches 100 ns; with the continuous development of new standards such as RoCE and iWARP, network connection technologies of different modes can be used in the search field to obtain better effects.
In addition, with the development of computing capabilities such as a non-volatile memory (NVM), a GPU, etc., in the future, the retrieval technology in a search engine can be further improved by combining the RDMA technology.
In order to implement the method according to the embodiment of the present invention, an embodiment of the present invention further provides a search apparatus, which is disposed on a search server, and as shown in fig. 9, the search apparatus includes:
an acquisition unit 91 for determining a keyword;
a first search unit 92, configured to search, from an index server, an index result matching the keyword based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode;
and the processing unit 93 is configured to perform sorting according to the index result to obtain a retrieval result. In an embodiment, the obtaining unit 91 is configured to determine a keyword;
searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on a Remote Direct Memory Access (RDMA) mode;
and sorting according to the index result to obtain a retrieval result.
In an embodiment, the first search unit 92 is configured to send a query request to the index server, where the query request includes the keyword; the query request is used for the index server to perform index query according to the key words;
and receiving an index result sent by the index server.
In one embodiment, the apparatus further comprises: the device comprises a first determining unit, a second determining unit and a control unit, wherein the first determining unit is used for determining a retrieval mode and determining a first target time length and a second target time length based on the retrieval mode;
the first target duration represents the duration of the index result; and the second target time length representation obtains the time length of the retrieval result based on the index result sorting.
In an embodiment, the first determining unit is specifically configured to determine the first target time duration based on a time duration required to search an index result meeting a first number of requirements from an index server when the retrieval mode is the first mode;
determining the second target duration based on a target delay duration and the first target duration; the target delay time represents the time allowed for retrieval.
In an embodiment, the first determining unit is specifically configured to determine the second target duration based on a duration required for sorting according to the index result when the retrieval mode is the second mode;
determining the first target duration based on a target delay duration and the second target duration; the target delay time represents the time allowed for retrieval.
In an embodiment, the first searching unit 92 is configured to search, within the first target duration, an index result matching the keyword from an index server based on the keyword.
In an embodiment, the first searching unit 92 is configured to perform sorting according to the index result within the second target duration to obtain a retrieval result.
In one embodiment, the apparatus further comprises: and the communication unit is used for establishing a communication relationship with the index server based on an RDMA protocol.
It should be noted that: in the searching apparatus provided in the above embodiment, only the division of the program modules is exemplified when performing the search, and in practical applications, the processing distribution may be completed by different program modules according to needs, that is, the internal structure of the apparatus may be divided into different program modules to complete all or part of the processing described above. In addition, the search apparatus and the search method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments in detail and are not described herein again.
Correspondingly, another searching device is provided, which is arranged on the index server and comprises:
a receiving unit, configured to receive a keyword sent by a search server;
the second searching unit is used for searching an index result matched with the keyword and sending the index result to the retrieval server; the index result is used for the search server to sort so as to obtain a search result;
wherein the search server communicates with the index server based on an RDMA approach.
Based on the hardware implementation of the program module shown in fig. 9, in order to implement the method according to the embodiment of the present invention, an embodiment of the present invention further provides a server. Fig. 10 is a schematic diagram of a hardware composition structure of a server according to an embodiment of the present invention, and as shown in fig. 10, the server 100 includes:
a communication interface 101 capable of performing information interaction with other devices such as network devices and the like;
and the processor 102 is connected with the communication interface 101 to realize information interaction with other devices, and is used for executing the method provided by one or more technical schemes on the search server side when running a computer program. And the computer program is stored on the memory 103.
Of course, in practice, the various components in the server 100 are coupled together by the bus system 104. It is understood that the bus system 104 is used to enable communications among the components. The bus system 104 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 104 in fig. 10.
The memory 103 in the embodiment of the present invention is used to store various types of data to support the operation of the server 100. Examples of such data include: any computer program for operating on the server 100.
It will be appreciated that the memory 103 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memory 102 described in connection with the embodiments of the invention is intended to comprise, without being limited to, these and any other suitable types of memory.
The method disclosed in the above embodiments of the present invention may be applied to the processor 102, or implemented by the processor 102. The processor 102 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 102. The processor 102 described above may be a general purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. Processor 102 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed by the embodiment of the invention can be directly implemented by a hardware decoding processor, or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in the memory 103, and the processor 102 reads the information in the memory 103 and performs the steps of the aforementioned methods in conjunction with its hardware.
When the processor 102 executes the program, the corresponding process implemented by the search server in each method according to the embodiment of the present invention is implemented, and for brevity, no further description is given here.
In an exemplary embodiment, the embodiment of the present invention further provides a storage medium, i.e. a computer storage medium, specifically a computer readable storage medium, for example, comprising a memory 103 storing a computer program, which is executable by a processor 102 of a search server to perform the steps of the foregoing method. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus, server and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or a part contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims (10)

1. A search method is applied to a retrieval server and comprises the following steps:
determining a keyword;
searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on a Remote Direct Memory Access (RDMA) mode;
and sorting according to the index result to obtain a retrieval result.
2. The method of claim 1, wherein the searching the index result matching the keyword from the index server based on the keyword comprises:
sending a query request to the index server, wherein the query request comprises the keyword; the query request is used for the index server to perform index query according to the key words;
and receiving an index result sent by the index server.
3. The method of claim 1, further comprising:
determining a retrieval mode, and determining a first target time length and a second target time length based on the retrieval mode;
the first target duration represents the duration of the index result; and the second target time length representation obtains the time length of the retrieval result based on the index result sorting.
4. The method of claim 3, wherein determining a first target duration and a second target duration based on the search pattern comprises:
when the retrieval mode is the first mode, determining the first target time length based on the time length required for searching the index result meeting the first quantity requirement from the index server;
determining the second target duration based on a target delay duration and the first target duration; the target delay time represents the time allowed for retrieval.
5. The method of claim 3, wherein determining a first target duration and a second target duration based on the search pattern comprises:
when the retrieval mode is a second mode, determining the second target time length based on the time length required for sorting according to the index result;
determining the first target duration based on a target delay duration and the second target duration; the target delay time represents the time allowed for retrieval.
6. The method according to claim 4 or 5, wherein the searching the index result matching the keyword from the index server based on the keyword comprises:
and searching an index result matched with the keyword from an index server based on the keyword in the first target time length.
7. The method according to claim 4 or 5, wherein the sorting according to the index result to obtain a retrieval result comprises:
and sequencing according to the index result within the second target time length to obtain a retrieval result.
8. A search apparatus provided on a search server, comprising:
an acquisition unit configured to determine a keyword;
the first searching unit is used for searching an index result matched with the keyword from an index server based on the keyword; the retrieval server is communicated with the index server based on an RDMA mode;
and the processing unit is used for sequencing according to the index result to obtain a retrieval result.
9. A server, comprising: a processor and a memory for storing a computer program capable of running on the processor,
wherein the processor is adapted to perform the steps of the method of any one of claims 1 to 7 when running the computer program.
10. A storage medium having a computer program stored thereon, the computer program, when being executed by a processor, performing the steps of the method of any one of claims 1 to 7.
CN202010451012.1A 2020-05-25 2020-05-25 Searching method, searching device, server and storage medium Active CN111625701B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010451012.1A CN111625701B (en) 2020-05-25 2020-05-25 Searching method, searching device, server and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010451012.1A CN111625701B (en) 2020-05-25 2020-05-25 Searching method, searching device, server and storage medium

Publications (2)

Publication Number Publication Date
CN111625701A true CN111625701A (en) 2020-09-04
CN111625701B CN111625701B (en) 2024-01-26

Family

ID=72259088

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010451012.1A Active CN111625701B (en) 2020-05-25 2020-05-25 Searching method, searching device, server and storage medium

Country Status (1)

Country Link
CN (1) CN111625701B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102033955A (en) * 2010-12-24 2011-04-27 常华 Method for expanding user search results and server
CN104915429A (en) * 2015-06-15 2015-09-16 小米科技有限责任公司 Keyword searching method and device
CN106909642A (en) * 2017-02-20 2017-06-30 中国银行股份有限公司 Database index method and system
CN108090153A (en) * 2017-12-11 2018-05-29 深圳云天励飞技术有限公司 A kind of searching method, device, electronic equipment and storage medium
CN108446389A (en) * 2018-03-22 2018-08-24 平安科技(深圳)有限公司 Speech message searching and displaying method, device, computer equipment and storage medium
CN109800348A (en) * 2018-12-12 2019-05-24 平安科技(深圳)有限公司 Search for information display method, device, storage medium and server
CN110399568A (en) * 2019-07-04 2019-11-01 Oppo广东移动通信有限公司 Information search method, device, terminal and storage medium
CN110659418A (en) * 2019-09-12 2020-01-07 北京达佳互联信息技术有限公司 Content searching method and device, storage medium and computing equipment

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102033955A (en) * 2010-12-24 2011-04-27 常华 Method for expanding user search results and server
CN104915429A (en) * 2015-06-15 2015-09-16 小米科技有限责任公司 Keyword searching method and device
CN106909642A (en) * 2017-02-20 2017-06-30 中国银行股份有限公司 Database index method and system
CN108090153A (en) * 2017-12-11 2018-05-29 深圳云天励飞技术有限公司 A kind of searching method, device, electronic equipment and storage medium
CN108446389A (en) * 2018-03-22 2018-08-24 平安科技(深圳)有限公司 Speech message searching and displaying method, device, computer equipment and storage medium
CN109800348A (en) * 2018-12-12 2019-05-24 平安科技(深圳)有限公司 Search for information display method, device, storage medium and server
CN110399568A (en) * 2019-07-04 2019-11-01 Oppo广东移动通信有限公司 Information search method, device, terminal and storage medium
CN110659418A (en) * 2019-09-12 2020-01-07 北京达佳互联信息技术有限公司 Content searching method and device, storage medium and computing equipment

Also Published As

Publication number Publication date
CN111625701B (en) 2024-01-26

Similar Documents

Publication Publication Date Title
US10831562B2 (en) Method and system for operating a data center by reducing an amount of data to be processed
US20160132541A1 (en) Efficient implementations for mapreduce systems
CA2925750C (en) An order book management device in a hardware platform
US20200136971A1 (en) Hash-table lookup with controlled latency
CN111459418B (en) RDMA (remote direct memory Access) -based key value storage system transmission method
US9535702B2 (en) Asset management device and method in a hardware platform
CN114201421B (en) Data stream processing method, storage control node and readable storage medium
CN108366018A (en) A kind of processing method of network data packets based on DPDK
Lockwood et al. Implementing ultra low latency data center services with programmable logic
CN111309805B (en) Data reading and writing method and device for database
Cacheda et al. Performance analysis of distributed information retrieval architectures using an improved network simulation model
US8694618B2 (en) Maximizing data transfer through multiple network devices
CN111625701B (en) Searching method, searching device, server and storage medium
CN111209263A (en) Data storage method, device, equipment and storage medium
Zhang et al. Design and implementation of a real-time interactive analytics system for large spatio-temporal data
WO2022142562A1 (en) Rdma-based communication method, node, system, and medium
CN111240845B (en) Data processing method, device and storage medium
Sumimoto et al. The design of advanced communication to reduce memory usage for exa-scale systems
CN107615259A (en) A kind of data processing method and system
Istvan Building Distributed Storage with Specialized Hardware
CN114328364B (en) Data transmission method, device and equipment in full text search engine and storage medium
Zhang et al. Communication optimization for RDMA-based science data transmission tools
CN116886605B (en) Stream table unloading system, method, equipment and storage medium
Mahanti et al. Controlled best-first search for near optimal solutions
Flynn et al. Overcoming Performance Bottlenecks With a Network File System in Solid State Drives

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant