KR102324571B1 - Method for providing enhanced search result in passage-based information retrieval - Google Patents

Abstract

Provided is a method for providing an improved search result when a document is divided into smaller sizes to generate a passage so as to be used as a search unit. A new ranking is generated by integrating the sorting result of the passage by a first search model and the sorting result of the passage by a second search model, and documents are sorted to correspond to the ranking. In addition, documents are sorted based on the relationship between the number of passages extracted from the same document and the total number of passages in the document. The final ranking is determined by merging the two sorting results for the document. According to the method, even when searching in units of passages, a search result with high relevance is generated by looking at the entire document.

Description

{Method for providing enhanced search result in passage-based information retrieval}

The present invention relates to a method of providing an improved search result in a passage-based search, and more specifically, dividing each document to be searched into a plurality of smaller-sized passages and storing them as a corpus, and storing these passages as a unit It relates to a method for solving problems that occur when performing searches and providing improved search results.

The technology for searching documents is a document-based search method in which a given document is searched as a whole, and a passage-based search method in which each document is divided into a plurality of smaller units to be searched. ) can be classified by the search method. Since each method has pros and cons, it is known that combining the search results using both methods together provides better results than when using either method individually. However, since the passage unit search has a disadvantage that it is necessary to divide and store each document into a plurality of passages, a document unit search method has been mainly adopted.

Recently, as the performance of the AI-based search model provides improved search results compared to the performance of the existing statistical-based search model, the passage unit search method is receiving attention again. Since the AI-based search model disclosed so far has a limit on the size of a document that can be processed, it is generally difficult to target the entire document. Accordingly, when an AI-based search model is applied, a passage-based search method is mainly adopted. For example, in the case of BERT, a language model published by Google, the maximum number of tokens that can be processed is limited to 512, so researchers divide documents into passages containing 100 to 300 tokens and process them as search targets. are doing

However, when a document is divided into passages to be a search target, a problem arises that does not appear when the entire document is a search target. For example, an example of a search result of the patent search system implemented by the present applicant is shown in FIGS. 7(a) to 7(c). As shown, when 'cosmetics' is entered as a query, the word 'cosmetics' appears twice in the document that ranked first in the search results (refer to FIG. 7(c)), and the passages including 'cosmetics' are total Only 2 out of 168 passages. On the other hand, the word 'cosmetics' appears a total of 122 times in the document appearing in 2nd place, and the passage containing 'cosmetics' is 61 out of 124 passages. Moreover, the document that ranked first contains content that is not related to the query 'cosmetics' as a whole. Therefore, if the entire document was searched, the search rankings of the two documents would be different.

WO 2016015267 A US 201000185623 A

An object of the present invention is to solve the above-mentioned problem, and to provide a search method for generating a search result with high relevance in view of the entire document even when searching in units of passages.

According to an aspect of the present invention, a search result corresponding to a query input by a user from a corpus including a plurality of passages extracted from each document of a document bundle composed of a plurality of search target documents is retrieved from the user. A computer-implemented method for providing to (b) rearranging the N passages based on the input query by a second search model; (c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages; (d) sorting M documents (M is less than or equal to N) including the N passages in a ranking corresponding to the combined ranking for the N passages; (e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages and the total number of passages in the corresponding document; and (f) integrating the sorting result in step (d) and the sorting result in step (e) for the M documents to determine the rank of the M documents. A computer-implemented method for

According to another aspect of the present invention, a search result corresponding to a query input by a user from a corpus including a plurality of passages extracted from each document of a document bundle consisting of a plurality of search target documents is retrieved by the user. An apparatus for providing to a user, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions stored in the at least one memory are executed by the at least one processor (a) by a first search model; extracting and sorting N passages corresponding to the input query from the corpus; (b) rearranging the N passages based on the input query by a second search model; (c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages; (d) sorting M documents (M is less than or equal to N) including the N passages in a ranking corresponding to the combined ranking for the N passages; (e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages and the total number of passages in the corresponding document; and (f) integrating the alignment result in step (d) and the alignment result in step (e) for the M documents so that the step of determining the rank of the M documents is executed. An apparatus for doing this is provided.

According to another aspect of the present invention, a computer program for providing a user with a search result corresponding to a query input by the user from a corpus comprising a plurality of passages, stored in a non-transitory storage medium, by a processor, (a) extracting and sorting N passages corresponding to the input query from the corpus by a first search model; (b) rearranging the N passages based on the input query by a second search model; (c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages; (d) sorting M documents (M is less than or equal to N) including the N passages in a ranking corresponding to the combined ranking for the N passages; (e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages and the total number of passages in the corresponding document; and (f) combining the alignment result in step (d) and the alignment result in step (e) for the M documents so that the step of determining the rank of the M documents is executed. There is provided a computer program stored in a non-transitory storage medium for retrieving a document to be used.

According to the present invention, an improved search result can be provided even when a search object document is divided into a plurality of passages to be searched.

1 is a configuration diagram for conceptually explaining a method for providing search results in units of passages as a method for providing search results according to an embodiment of the present invention;
2 is a flowchart of a method for providing a search result based on a passage unit search result according to an embodiment of the present invention;
Fig. 3 schematically shows an apparatus for carrying out the method shown in Fig. 2;
4 is a configuration diagram for conceptually explaining a method for determining a ranking by integrating a passage unit search result and a document unit search result as a search result providing method according to another embodiment of the present invention;
5 is a flowchart of a method for providing a search result by integrating a passage unit search result and a document unit search result according to an embodiment of the present invention;
Fig. 6 schematically shows an apparatus for carrying out the method shown in Fig. 5;
7(a) to 7(c) are diagrams for explaining a problem of a passage unit search method.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings. The same or similar reference numerals are assigned to the same or similar components, and overlapping descriptions thereof are omitted. In describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. The accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It should be understood to include what is or replaces.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but these terms are used only for the purpose of distinguishing one element from other elements, and the corresponding elements are defined by these terms. not limited by The singular expression includes the plural expression unless the context clearly dictates otherwise.

As used herein, terms such as "comprises", "comprises" or "have" are to be understood as limiting the existence of a feature, step, element, or combination thereof described in the specification, and one or more other elements. It is not intended to exclude the possibility that features, steps, components, or combinations thereof may be present or added.

1 to 3 are views according to an embodiment of the present invention. This embodiment relates to the integration of search results using two or more passage unit search models. In the illustrated embodiment, two search models are shown, but three or more search models may be used.

The search system 100 includes a document bundle 112 in which documents are stored, and a corpus 114 (corpus) in which each document of the document bundle is divided into a plurality of smaller units and stored. Although not shown in the drawing, information about the correspondence relationship of the document-passage, the number of passages included in each document, and the like is additionally stored.

The first search model 122 extracts and sorts N passages from the corpus based on a query input by the user. The second search model 124 rearranges the N passages extracted by the first search model 122 based on a query input by the user. Although the second search model 124 may extract and sort passages from the corpus independently of the first search model 122 , in this embodiment, the method as described above is adopted. This method has recently been widely used to solve the problem that AI-based search models have high precision but slow processing speed. That is, a passage is first extracted from the corpus by using the first search model 122 with a high processing speed, and the second search based on artificial intelligence has a relatively low processing speed but relatively higher accuracy than the first search model 122 . By using the model 124 and rearranging the passages extracted by the first search model 122, the processing speed and accuracy are compromised. In this case, the first search model 122 preferably has a high recall. As the first search model 122, a search model based on statistical techniques, for example, a search model such as BM25, is mainly used. ) faster search models are being proposed. In summary, in the configuration as in the present embodiment, it is preferable to use the first search model 122 having a relatively high recall rate and faster processing speed than the second search model 124 , and the second search model 124 . is relatively high in accuracy and the processing speed may be slower than that of the first search model 122 .

_{The alignment result (r1 p} ) by the first search model 122 and the rearrangement result (r2 _p ) by the second search model 124 for the N passages are rank-combined 132, and the integrated alignment result (rf _p) ) is provided. Rank fusion is a technique for providing a single search result by integrating search results by different search models, and various integration methods have been proposed. A representative method is reciprocal rank fusion (RRF). Rank combining is not just about integrating sorting results, but is generally known to improve the accuracy of search results.

_{The sorting result (rf p} ) for the N passages integrated in the rank combination 132 is _{the rank (r1 d} , r2 _d ) of the document corresponding to the passage through the aggregation 134 and the Np~np relationship analysis 136. each is converted.

Aggregation means processing a case in which two or more passages included in one document exist among N passages. All of the N passages extracted by the first search model may be included in different documents, or some passages may be included in the same document. Therefore, if the number of documents corresponding to N passages is M, a relationship of N≥M is established. Accordingly, when there are two or more passages extracted from the same document among the N passages extracted by the first search model, there is a problem of which passage is to be determined as the priority of the corresponding document. In this regard, various methods have been proposed, for example, a method of determining the ranking of passages having the highest search similarity as the ranking of corresponding documents, a method of summing the search similarity of all passages included in a document, a document There is a method of determining the search similarity of the first passage of . In this embodiment, there is no guarantee that all passages included in a specific document are included in the N passages extracted by the first search model 122, and there is no guarantee that the first passage of the document is extracted. Therefore, when there are two or more passages extracted from the corresponding document, the second and third methods are difficult to adopt, and the ranking of the passage having the highest search similarity among these passages is determined as _{the ranking (r1 d) of the corresponding document.} However, it is not necessarily limited to this method, and, for example, when there are a plurality of passages extracted from the same document among N passages, the search similarity of the corresponding document may be determined by appropriately summing the search similarities. For example, Kong et al. in "Kong, K., et al. "Passage-based retrieval using parameterized fuzzy set operators." ACM SIGIR Workshop on Mathematical/Formal. Methods for Information Retrieval. 2004." Three methods have been proposed: Disjunctive Relevance Decision (DRD), Aggregate Relevance (AR), and Conjunctive Relevance Decision (CRD).

Np ~ np relationship analysis 136 analyzes the relationship between the total number of passages Np of a specific document and the number of passages np included in the N passages extracted by the first search model from among passages of the corresponding document by analyzing the relationship between the N passages It means to determine _{the rank (r2 d} ) of the document corresponding to . For example, if the number of documents corresponding to N passages is M, for these M documents, the documents with the smallest value given by the relation of (Np-np)/Np are sorted in the order of the documents with the largest value. can be Conversely, documents having a large value given by the relational expression of np/Np may be sorted in the order of documents having a small corresponding value. When examining the case shown in FIGS. 7(a) to 7(c), in the case of the document shown in the first place, the total number of passages (Np) is 168, and the number of passages that would have been extracted by the first search model (np) ) is 2, so (Np-np)/Np = (168-2)/168 = 0.9881 and np/Np = 2/168 = 0.0119, and for the second-placed document, the total number of passages (Np) is Since there are 124 and the number of passages to be extracted (np) is 61, (Np-np)/Np = (124-61)/124 = 0.5081 and np/Np = 61/124 = 0.4919. Accordingly, when two documents are ranked according to the first method, the priority of document 2 is higher than that of document 1, and even if the ranking is determined by the second method, the priority of document 2 is higher than that of document 1.

_{The rank (r1 d} ) of the document determined by the aggregation 134 _{and the rank (r2 d} ) of the document determined by the Np~np relationship analysis 136 are combined by the rank combination 138 . The same technique as the rank combining 132 for passages may be used or a different technique may be used. 7(a) to 7(c), if the document ranking by the Np~np relationship analysis 136 is integrated, the ranking between the document 1 and the document 2 may be changed.

_{The rank (rf d} ) of the integrated document by the rank combination 138 is provided to the user as a final search result.

The method according to this embodiment will be described with reference to FIG. 2 as follows: First, N passages corresponding to the query input from the corpus 114 are extracted and sorted by the first search model 122 ( S110). Next, the N passages extracted by the first search model 122 are rearranged based on the query input by the second search model 124 ( S120 ). In general, the sorting results by the two models are different. Is generated N sorted at step S110 for the two passages (r1 _p) and rearranged at step S120 (r2 _p) the integrated (132) to the integrated ranking (rf _p) for the N passage (S130) . The merging of the sorting result and the reordering result may be achieved by a reciprocal rank combining (RRF) method. M documents (M is less than or equal to N) including N passages are sorted (134) with a ranking corresponding to the unified ranking for the N passages (S140), and extracted from the same document among the N passages. The M documents are arranged 136 based on the relationship between the number of passages and the total number of passages of the corresponding document (S150). And ranking (rf _d) for the M document determined by integrating the results of the sort (r2 _d) in the sorting in step S140 for the M document result (r1 _d) and step S150, In this way, the final search results, is provided to the user.

3 shows a computer device for performing the search method according to the present embodiment. Since the method according to the present embodiment has been described in detail with reference to FIGS. 1 and 2 , an apparatus 300 for performing such a method will be schematically described with reference to FIG. 3 .

Referring to FIG. 3 , the computer device 300 includes a processor 310 , a non-volatile storage unit 320 for storing programs and data, a volatile memory 330 for storing programs being executed, and information between a user and a user. It consists of an input/output unit 340 for inputting and outputting , and a bus, which is an internal communication path between these devices. The running program may include an operating system and various applications. Although not shown, it includes a power supply unit.

The first search model 222 and the second search model 224 stored in the storage unit 320 are executed in the memory 330 . The corpus 114 is stored in the storage unit 320 , and a search method is performed based on a query input through the input/output unit 340 . The storage unit 320 additionally stores information on correspondence between documents and passages, the number of passages included in each document, and the like. The document-passage correspondence is used in step S140 for the aggregation 134 , and the number of passages included in each document is used in step S150 for the Np-np relationship analysis 136 .

4 to 6 show another embodiment of the present invention. In the above embodiment, a method for providing search results of corresponding documents by integrating the search results of two or more passage unit search models has been described, but in this embodiment, the ranking is obtained by integrating the passage unit search results and the document unit search results. A method for determining is described.

The search system 400 includes a document bundle 112 in which documents are stored, and a corpus 114 (corpus) in which each document of the document bundle is divided into a plurality of smaller units and stored. Although not shown in the drawing, information about the correspondence relationship of the document-passage, the number of passages included in each document, and the like is additionally stored.

D documents corresponding to the query input by the user are extracted and sorted from the document bundle 112 by the document unit search model 410 . As the document unit search model 410, a model capable of performing a search for the entire document is used. For example, BM25 may be used as a statistical technique. Recently, a method of improving performance by replacing the indexing file based on the TF-IDF of the BM25 with another indexing file generated using an artificial intelligence technique has been proposed.

N passages corresponding to the query input by the user from the corpus 114 are extracted and sorted by the passage unit search model 420 . The passage unit search model 420 may consist of one search model, or two or more search models may be used as in the above-described embodiment. Since the case of using two or more search models has been described in detail in the above embodiment, only the case of using one search model will be described in this embodiment. After N passages are extracted and sorted by the passage unit search model 420, M documents (M is less than or equal to N) containing N passages through the aggregation 134 are sorted by N passages. They are sorted by rank corresponding to the rank. In addition, the M documents are sorted based on the relationship between the number of passages extracted from the same document among the N passages and the total number of passages of the corresponding document ( 136 ).

Sort result of the sort result by the document unit search model (410), (r3 _d), sort result by the aggregate 134 related to the passage unit search (r1 _d) and Np ~ np Relationship (136) (r2 _d) are rank-combined 138 and provided as a final search result. As a rank combining technique, a rank combining is possible even if there is no search result of any one search model. Accordingly, even if D, the number of documents extracted and sorted by the document unit search model 410, and M, the number of documents corresponding to the N passages extracted by the passage unit search model 420, are not the same, the rank combination is not possible. For example, a reciprocal rank combining (RRF) scheme may be used.

The method according to this embodiment will be described with reference to FIG. 5 as follows: First, D documents corresponding to the query input by the user are extracted from the document bundle 112 by the document unit search module 410 and aligned (S210). In addition, N passages corresponding to the query input from the corpus 114 are extracted and sorted by the passage unit search module 420 (S220). M documents (M is less than or equal to N) including N passages are sorted (S230) in a rank corresponding to the rank in which the N passages are sorted, the number of passages extracted from the same document among the N passages and Based on the relationship between the total number of passages of the document, M documents are sorted (S240). Finally, the order of documents to be provided as a search result is determined by integrating the alignment results of the D documents in step S210 and the alignment results in step S230 with the alignment results in step S240 for the M documents.

6 shows a computer device for performing the search method according to the present embodiment. Since the method according to the present embodiment has been described in detail with reference to FIGS. 4 and 5 , an apparatus 300 for performing such a method will be schematically described with reference to FIG. 6 .

Referring to FIG. 6 , the computer device 300 includes a processor 310 , a non-volatile storage unit 320 for storing programs and data, a volatile memory 330 for storing programs being executed, and information between a user and a user. It consists of an input/output unit 340 for inputting and outputting , and a bus, which is an internal communication path between these devices. The running program may include an operating system and various applications. Although not shown, it includes a power supply unit.

The document unit search model 410 and the passage unit search model 420 stored in the storage unit 320 are executed in the memory 330 . The document corpus 112 and the corpus 114 are stored in the storage unit 320 , and a search method is performed based on a query input through the input/output unit 340 . The storage unit 320 additionally stores information on correspondence between documents and passages, the number of passages included in each document, and the like. The document-passage correspondence is used in step S230 for the aggregation 134 , and the number of passages included in each document is used in step S240 for the Np-np relationship analysis 136 .

The embodiment according to the present invention shown in Figs. 4 to 6 can be used when combining a new AI-based passage unit search system with an existing document unit search system.

The foregoing detailed description should not be construed as restrictive in any way but as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (11)

A computer implemented for providing a user with a search result corresponding to a query input by the user from a corpus including a plurality of passages extracted from each document of a document bundle consisting of a plurality of search target documents As a method,
(a) extracting and sorting N passages corresponding to the input query from the corpus by a first search model;
(b) rearranging the N passages based on the input query by a second search model;
(c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages;
(d) sorting M documents (M is less than or equal to N) including the N passages in a rank corresponding to the combined rank for the N passages;
(e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages (np) and the total number of passages of the corresponding document (Np); and
(f) determining the ranking of the M documents by integrating the alignment results in step (d) and the alignment results in step (e) for the M documents
including,
In step (e), the M documents are arranged in the order from a document having a small value given by the relation of (Np-np)/Np to a document having a large corresponding value, or having a large value given by the relation of np/Np. Documents are sorted in descending order from document to document.
A computer-implemented method for retrieving documents.
The method according to claim 1,
The integration of the alignment results in step (c) and step (f) is made in a reciprocal rank fusion (RRF) method.
A computer-implemented method for retrieving a document, characterized in that
delete delete An apparatus for providing a user with a search result corresponding to a query input by the user from a corpus including a plurality of passages extracted from each document of a document bundle consisting of a plurality of search target documents, the apparatus comprising:
at least one processor; and
at least one memory for storing computer-executable instructions;
The computer-executable instructions stored in the at least one memory are executed by the at least one processor,
(a) extracting and sorting N passages corresponding to the input query from the corpus by a first search model;
(b) rearranging the N passages based on the input query by a second search model;
(c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages;
(d) sorting M documents (M is less than or equal to N) including the N passages in a rank corresponding to the combined rank for the N passages;
(e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages (np) and the total number of passages of the corresponding document (Np); and
(f) determining the ranking of the M documents by integrating the alignment results in step (d) and the alignment results in step (e) for the M documents
to run, but
In step (e), the M documents are sorted in the order from a document having a small value given by the relation of (Np-np)/Np to a document having a large value, or having a large value given by the relation of np/Np. Documents are sorted in descending order from document to document.
A device for retrieving documents.
6. The method of claim 5,
At least some of the documents in the document bundle have a title,
Each passage of the corpus includes the title of a document in which the passage is included as a part
A device for retrieving documents, characterized in that.
delete delete 6. The method of claim 5,
The integration of the alignment results in step (c) and step (f) is made in a reciprocal rank fusion (RRF) method.
A computer-implemented device for retrieving documents, characterized in that
6. The method of claim 5,
At least one of the first search model and the second search model is an artificial intelligence-based search model.
A device for retrieving documents, characterized in that.
A computer program for providing a user with a search result corresponding to a query input by the user from a corpus comprising a plurality of passages, the computer program comprising:
It is stored in a non-transitory storage medium, and by the processor,
(a) extracting and sorting N passages corresponding to the input query from the corpus by a first search model;
(b) rearranging the N passages based on the input query by a second search model;
(c) generating an integrated ranking for the N passages by integrating the alignment result in step (a) and the reordering result in step (b) for the N passages;
(d) sorting M documents (M is less than or equal to N) including the N passages in a rank corresponding to the combined rank for the N passages;
(e) sorting the M documents based on a relationship between the number of passages extracted from the same document among the N passages (np) and the total number of passages of the corresponding document (Np); and
(f) determining the ranking of the M documents by integrating the alignment results in step (d) and the alignment results in step (e) for the M documents
contains a command that causes it to be executed,
In step (e), the M documents are arranged in the order from a document having a small value given by the relation of (Np-np)/Np to a document having a large corresponding value, or having a large value given by the relation of np/Np. Documents are sorted in descending order from document to document.
A computer program stored on a non-transitory storage medium for retrieving documents.

Images