KR102216066B1 - Method for providing search result for query in a sentence - Google Patents

Abstract

Provided is a method for providing a search result for a sentence-type query using an artificial intelligence-based search model. A sentence-type query is converted into a keyword query through sequence labeling. The keyword query may include keywords within a predetermined range, and stop words included in the sentence-type query may be removed. The artificial intelligence-based search model provides appropriate search results, even when the number of keywords included in the keyword query is insufficient or too many.

Description

{Method for providing search result for query in a sentence}

The present invention relates to a method of providing a search result for a sentence-type query input, and more specifically, providing a search result for a query having a sentence form using an information retrieval model learned based on the WickSupervision methodology. It's about how.

Search technology has been continuously developed since Google introduced the PageRank search technology based on graph theory. This search technique was based on unsupervised learning, and it was possible to search only given a bunch of documents. BM25 is a representative search model based on unsupervised learning, and it shows very improved performance when used with a query expansion technique called RM3. As an open source, Anserini is widely used in the academic field and in the field. In the search model based on such unsupervised learning, query input is mainly in the form of keywords.

Meanwhile, in the field of natural language processing, various search models have been proposed in accordance with research in the academic field to apply artificial intelligence techniques. For example, deep learning-based search models such as DRMM, KNRM, and PACRR have been proposed. BERT, released by Google in 2018, has shown good performance in various natural language processing fields, and research has been conducted to use it as a search model based on a transformer or language model. The artificial intelligence-based search model can process not only keyword queries but also sentence-type queries.

In the Ad-Hoc Information Retrieval section of Paper With Code, a website that introduces artificial intelligence models that are open source in each field, the current point of view of artificial intelligence-based search models including Anserini, a search model based on unsupervised learning. SOTA (State-of-the-Art), that is, the search model that shows the best performance can be identified. According to a researcher at the University of Waterloo, Canada, named Jimmy Lin, deep learning search models before BERT, i.e., DRMM, KNRM, and PACRR, are similar to Anserini, a search model based on unsupervised learning methodology. Although the performance is similar or worse, the models proposed after BERT are said to have improved performance over Anserini (see Lin, Jimmy. "The Neural Hype, Justified! A Recantation."). Such matters can also be checked on the leader board of the Ad-Hoc Information Retrieval item of the above Paper With Code. From these academic research results, it can be seen that the accuracy of the search results can be improved by the search model based on artificial intelligence.

However, artificial intelligence-based search models have some limitations.

In order to use an artificial intelligence-based search model for inference, it must first be trained, and such learning requires a large amount of labeled data. Labeled data must be basically processed and provided by humans, but it is uneconomical because the cost for labeling is too high when considering the amount of data required for learning. As described above, the artificial intelligence-based search model can process sentence-type queries as inputs, but it is not easy to prepare them when there is no learning data. A data set is provided for academic research, but depending on the field, it may not be enough to just learn it with the data for the purpose of academic research.In order to learn an artificial intelligence-based search model, not only positive data but also negative data are required. , There is a limitation in that the above data set only provides positive data.

As another problem, search models based on unsupervised learning generally do not matter even if the length of the document is long, but most of the search models based on artificial intelligence have limitations in the length of documents that can be processed. In the case of BERT, for example, the maximum number of tokens that can be processed is limited to 512. Therefore, there is no problem in the case of searching for a corpus consisting of short articles, but there is difficulty in applying it in the case of searching for long documents such as patents and papers.

[1] https://paperswithcode.com/task/ad-hoc-information-retrieval [2] MacAvaney, Sean, et al. "CEDR: Contextualized embeddings for document ranking." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019. [3] Dai, Zhuyun, and Jamie Callan. "Deeper text understanding for IR with contextual neural language modeling." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.

The present invention aims to solve the above-described problem, and provides a method of providing an effective search result in a case where a sentence-type query is given as an input using an artificial intelligence-based search model.

According to an aspect of the present invention, as a computer-implemented method for providing a search result corresponding to a sentence-type query from a corpus including a plurality of passages, (a) a sentence through sequence labeling Converting the type query into a keyword query; (b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology; (c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And, (d) outputting a list of search results corresponding to the N messages re-ranked in step (c). A computer-implemented method for searching a document is provided.

According to another aspect of the present invention, an apparatus for providing a search result corresponding to a sentence type query from a corpus including a plurality of passages, comprising: at least one processor; And at least one memory for storing a computer-executable instruction, wherein the computer-executable instruction stored in the at least one memory, by the at least one processor, (a) a sentence form through sequence labeling. Converting the query into a keyword query; (b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology; (c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And (d) a step of outputting a list of search results corresponding to the N messages re-ranked in step (c) is provided.

According to another aspect of the present invention, a computer program for providing a search result corresponding to a sentence-type query from a corpus including a plurality of passages, stored in a non-transitory storage medium, by a processor, (a) a sequence Converting a sentence type query into a keyword query through labeling; (b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology; (c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And, (d) a computer program stored in a non-transitory storage medium for searching for a document including an instruction to execute a step of outputting a list of search results corresponding to the N messages reranked in step (c). Is provided.

According to the present invention, there is provided a method of providing an effective search result for a case where a sentence-type query is given as an input using an artificial intelligence-based search model.

1 is a flow chart showing a method for learning an information retrieval model based on artificial intelligence according to the present invention.
2 is a flowchart showing an information retrieval method using an artificial intelligence-based information retrieval model according to the present invention.
3 is a diagram schematically showing an information retrieval model according to an example of the present invention.
4 is a diagram schematically showing an apparatus for performing an information retrieval method using an information retrieval model based on artificial intelligence according to the present invention.
5 is a diagram for comparing a search result with a search result of another model according to an embodiment of the present invention.
6 is a view comparing a search result according to an embodiment of the present invention with a search result of another model.
7 is a diagram schematically showing a search model using a sentence-type query as an input according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same or similar reference numerals are assigned to the same or similar components, and redundant descriptions thereof are omitted. In describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. The accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It is to be understood as including or substituting.

Terms including ordinal numbers, such as first and second, can be used to describe various elements, but these terms are used only for the purpose of distinguishing one element from other elements, and the corresponding elements are these terms. Is not limited by Singular expressions include plural expressions unless the context clearly indicates otherwise.

As used herein, terms such as "comprises", "includes" or "have" should be understood as limiting the existence of features, steps, components, or combinations thereof described in the specification. It is not intended to exclude the possibility of the presence or addition of features, steps, components, or combinations thereof.

Fig. 1 shows a method of learning an artificial intelligence-based IR model according to the present invention, and Fig. 2 shows a step of performing inference using the artificial intelligence-based IR model according to the present invention, 3 schematically shows an example of an information retrieval model according to the present invention. Hereinafter, the present invention will be described with reference to these drawings.

[Learning stage of information retrieval model based on artificial intelligence]

Information retrieval model

In this specification, the information retrieval model is classified into'a retrieval model based on unsupervised methodology' and'a retrieval model based on artificial intelligence'. The former refers to an information retrieval model based on statistical or other unsupervised learning methodologies such as BM25 and QL (Query Likelihood), and the latter is a deep learning series such as DRMM, KNRM, and PACRR, and a language model series such as BERT. It refers to an information retrieval model prepared by learning including, etc.

In the present invention, the retrieval model based on the former unsupervised learning methodology is used to prepare a pseudo-label in the learning stage, and in the inference stage to retrieve a passage from the corpus, and the latter It is used for re-ranking the retrieved passages in the inference stage.

Information retrieval models with open sources are disclosed in the Ad-Hoc Information Retrieval section of Paper With Code.

Search model based on unsupervised learning methodology

As a search model based on an unsupervised methodology, BM25 and QL (Query Likelihood) are known, and Anserini is widely used as an open source. Anserini is a search model that uses RM3, a query expansion methodology, along with BM25. According to research results, the performance of Anserini is similar to that of deep learning search models such as DRMM, KNRM, and PACRR, but it is known to be inferior to that of language model series such as BERT.

The search model based on the unsupervised learning methodology does not require learning and determines the similarity between the query and the document based on statistical theories.

Artificial intelligence-based search model

As a search model based on artificial intelligence, a search model based on deep learning such as DRMM, KNRM, and PACRR, and a search model based on a transformer or language model such as BERT are known. According to the Paper With Code, as of the filing date of this patent, the SOTA (State-of-the-Art) search model has a structure that combines the BERT of the language model and the search model of the deep learning system. It is understood to be.

The artificial intelligence-based search model is completed by learning the similarity between a query and a document based on learning data. In academic terms, AI-based search models are trained using datasets that provide query-document relationships such as TREC, SQuAD, and MS Marco.

Language model search engines such as BERT use Self-supervision learning methodology in the pre-training stage, but deep learning in the fine-tuning stage for information retrieval. Like the search model of, supervised learning based on the data set for the query-document relationship is required.

Weaksupervision methodology

Query-document relational datasets such as TREC, SQuAD, and MS Marco, which are used academically, correspond to ground truths extracted based on actual user search activities, or some use the title of a document as a query. However, there is only a document corpus in the actual field, and there is no query input by the user to search for it, or even if it exists, it is very insufficient to learn an artificial intelligence-based search model. In the present invention, when only a document bundle exists, a pseudo-query is generated from each document, a pseudo-query-document relationship is formed using the generated pseudo-query, and a pseudo-label is generated. In addition, a methodology for learning an artificial intelligence-based search model using this is provided. The WickSupervision methodology for an artificial intelligence-based search model according to the present invention includes the following steps: 1) generating a pseudo-query from each document in a document bundle, 2) the generated pseudo-query Creating a pseudo-query-document relationship by using and generating a pseudo-label based on the pseudo-query-document relationship, 3) Learning an artificial intelligence-based search model using the generated pseudo-label . Hereinafter, each of the above-described steps will be described separately.

1) creating a pseudo-query

One or more keywords are extracted from each document in the document bundle, and this is determined as a pseudo-query. A method of extracting keywords from documents uses a previously known keyword extraction technique. Keyword extraction algorithms can be largely divided into techniques based on unsupervised learning and techniques based on supervised learning, and there are open sources for a number of techniques.

Most of the techniques are methods of extracting a plurality of keywords or phrases from a document, but recently, a technique called Doc2Query, which creates a natural language sentence-type query using BERT as a supervised learning-based technique, has also been released as an open source. There is a bar.

2) creating a pseudo-label

The pseudo-query extracted from each document is used as an input of a search model based on unsupervised learning methodology such as BM25, and M documents are retrieved from the document stack. In this case, the document from which the pseudo-query is extracted is likely to be included in the upper rank among the M documents, but may not be included.

Among the retrieved M documents, top m (<M) documents are labeled as positive training data, and at least some of the remaining M documents are labeled as negative training data. Depending on the AI-based search model, additional parts can be labeled as neutral training data. In general, positive data and negative data are required, but neutral data is not.

Here, the number of documents M to be retrieved, the number of positive training data m, the number of negative training data, and the number of neutral training data are integers, and may be differently determined according to the characteristics of the document bundle as a kind of hyper parameter. Developers can increase the accuracy of artificial intelligence-based search models by adjusting these hyperparameters in an experimental or theoretical approach according to the characteristics of the document bundle.

3) Learning an artificial intelligence-based search model

Since a query-document relationship is required to train an artificial intelligence-based search model, the training data is provided in the form of a relationship between a query used for retrieving and a document retrieved by the query. For example, it may be given in the form of a pair of two data such as (pseudo-query, positive document), or three data may be provided in a related form such as (pseudo-query, positive document, negative document). Of course, up to a neutral document may be associated with four data.

The amount of data required for learning is not known exactly, and is mainly confirmed by experimental methods. However, the amount of data required for training increases according to the number of parameters included in the model. BERT-series language model-based search models generally have pre-learned models based on Wikipedia, and fine-tuning pre-trained models for specific tasks such as information retrieval is compared with prior learning. Therefore, a small amount of data is required.

In academic research, the amount of learning data is determined according to the available datasets, but in the actual field, the learning data can be determined according to the amount of documents included in the document stack. If the amount of documents included in the document stack is insufficient for learning, another document of a similar field can be added to the document stack, or the amount of data can be increased by using a data augmentation technique.

When training an artificial intelligence-based model, it is generally desirable to divide the data into training data and verification data to verify performance. For example, 80% of the data can be used for training and the remaining 20% of the data can be used for verification.

[Inference stage]

In the present invention, the reasoning step refers to a process of providing a search result in response to a query input by a user. The reasoning step can be broadly divided into the following steps: a retrieving step, a re-ranking step, and an output step. Hereinafter, each step will be described in detail.

Retrieving stage

This is a step of extracting N documents from a bunch of documents to be searched by a search model based on an unsupervised learning methodology. In general, the search model based on artificial intelligence takes considerably longer time for inference than the search model based on unsupervised learning methodology. Therefore, when an artificial intelligence-based search model is applied to all documents included in the document stack, it takes an excessively long time and the user's convenience is degraded. Therefore, after retrieving a number of documents by a search model based on an unsupervised learning methodology, which is relatively fast in the first lane, it is common to apply an artificial intelligence-based search model only to the second retrieved documents. Research to skip the retrieving step by improving the inference speed of the artificial intelligence-based search model is continuously being conducted, but it is understood that it has not yet reached a sufficiently fast inference speed in consideration of user convenience.

At this stage, accuracy is also important, but recall is more important. On the other hand, in the second re-ranking step, accuracy is more important than recall. One of the ways to increase the recall rate is to retrieve the number of documents several times larger than the number of documents to be provided in the search result. Another solution is to increase the reproducibility of the search model based on the unsupervised learning methodology. By using a technique called DeepCT Index using BERT, it is possible to increase the reproducibility while using a search model based on the same unsupervised learning methodology. In order to retrieve a large amount of documents, the time required for retrieving increases in proportion to the number of documents. In the case of using the DeepCT Index technique, it is possible to achieve the same level of reproducibility as in the case of retrieving several times more documents while reducing the time required for searching by reducing the number of documents to be retrieved.

As a search model based on the unsupervised learning methodology, the same search model as used in the learning stage may be used, or another search model may be used. It is important to select a search model with a high reproducibility in the field. In academic research, Anserini (BM25+RM3) is widely used as an open source based search model.

Re-ranking stage

The relevance between the retrieved N documents and the query entered by the user is evaluated by a search model based on the unsupervised learning methodology and sorted according to the degree of relevance. In this step, the relationship between the retrieved N documents and the query entered by the user is re-evaluated by the artificial intelligence-based search model, and then re-ordered, that is, re-ranked. This step is performed by the artificial intelligence-based search model learned in the above-described learning step.

Search results output

The N documents re-ranked by the artificial intelligence-based search model are sorted according to the order of the degree of relevance and output as a search result.

[Processing in special cases]

Matters related to the length of the document

In general, a search model based on unsupervised learning methodology does not have any problem in execution even if the document length is long, but the search model based on artificial intelligence often has a limit on the maximum length of a document that can be processed. In particular, AI-based search models introduced after BERT are limited in the number of maximum tokens that can be processed. For example, in the case of BERT, the maximum number of tokens that can be processed is limited to 512.

When there is a limit on the maximum length of documents that can be processed by the artificial intelligence-based search model, each document in the document corpus can be divided into passages less than the limited length and used as a search target. In this specification, a bundle of passages made of short texts is expressed as a'corpus' in order to distinguish it from a document pile made of a long document.

In the learning stage, it is the same as described above, except that the passages included in the corpus are replaced with documents. That is, a pseudo-query is extracted from a passage, a pseudo-label is created using a search model based on an unsupervised learning methodology, and then an artificial intelligence-based search model is trained. The artificial intelligence-based search model is learned based on the relationship between the pseudo-query and the passage, not the relationship between the pseudo-query and the entire document.

In the inference stage, it is necessary to pre-process each document of the document stack into a corpus by dividing it into a passage, and storing the correspondence between the passage and the document in a form that can be referenced. In the retrieving step, N passages are retrieved from the corpus based on a query input by the user, and in the re-ranking step, the retrieved N passages are reranked.

In the search results, documents other than passages must be provided in order of relevance. To this end, documents are sorted so as to correspond to the sorting order of the re-ranked passage by referring to the correspondence relationship between the passage provided in the preprocessing step and the document, and provided as a search result. Since several passages are separated from one document, the reranked result may include a plurality of passages extracted from one document. In this case, the sorting order of documents may correspond to the order of the most relevant passages.

Document expansion

A long document contains multiple paragraphs, but the content of each paragraph does not necessarily match the subject. Therefore, when each paragraph is divided into passages and stored as a corpus, there is a possibility that the degree of relevance may be evaluated low even though the paragraph is separated from the relevant document.

To prevent this problem, the title of the document or equivalent text can be added to each separate passage in the document. In other words, the same phrase is added to each separate passage in a document. This expansion by adding a phrase having a relevance to the passage is expressed in the term'document expansion' in this specification. The phrase corresponding to the title of the document can be generated, for example, using a text summarization technique. The document summarization technique is largely divided into extractive summarization and abstract summarization, and both can be applied. Techniques for generating additional phrases for document expansion are not necessarily limited to the document summary technique, and any technique capable of compressively expressing the subject of the document may be used.

For example, a query entered by a user includes a plurality of keywords, some of which are included in the original passage, but other keywords may be included in a phrase added by document expansion. Since the text added by the document expansion includes the subject of the entire document, there is a possibility that the passage contains the content intended to be searched by the user. If the text was not added by document expansion, a message that would not have been retrieved can be retrieved by document expansion. This can increase the recall of the retrieving step. As described above, it is important to have high reproducibility in the retrieving step.

If the length of the text added to the passage by document expansion is too long, the length of the passage that is increased by document expansion may exceed the length that can be processed by the artificial intelligence-based search model. In order to minimize the side effects resulting from this, it is desirable that the text added by document expansion be placed in front of the passage. This is because even if the trailing end of a passage lengthened by document expansion is cut off, if the keyword included in the query entered by the user is not included in the trailing end, the passage will still be extracted in the retrieving step.

Domain adaptiveness

When an artificial intelligence-based search model is trained as a document of a specific domain, it can be used as a search model specialized for that domain. For example, in the case of fine-tuning the BERT pre-learned as a general document bundle such as Wikipedia to a document of a specific domain, in the pre-learning stage, the general relationship between the vocabularies is learned, and in fine-tuning, the vocabulary specialized for the domain is learned. You will learn.

A search model specialized for a specific domain may have relatively poor performance for other domains, but it improves performance for that domain.

Ensemble search model

In the above method, the results re-ranked by the artificial intelligence-based search model were used as the final search results, but the search results were provided using an ensemble of search models based on unsupervised learning methodology and search models based on artificial intelligence. May be. In this case, the final evaluation is expressed as in Equation 1.

[Equation 1]

(Final evaluation) = a*(Evaluation of search model based on unsupervised learning methodology)

+ (1-a)*(Evaluation of artificial intelligence-based search model)

In Equation 1, a is a value between 0 and 1, and may be adjusted to provide the best search result as a hyper parameter.

The ensemble model does not necessarily guarantee that the performance is improved compared to the single model, and it can be considered whether or not it is adopted depending on the domain.

4 shows a computer device for performing a search method according to the present invention.

Since the search method and the learning method according to the present invention have been described in detail with reference to FIGS. 1 to 3, an apparatus 100 for performing such a search method will be briefly described with reference to FIG. 4.

Referring to FIG. 4, the computer device 100 includes a processor 110, a nonvolatile storage unit 120 storing programs and data, a volatile memory 130 storing programs being executed, and information between a user and a user. It consists of an input/output unit 140 for inputting and outputting, and a bus, which is an internal communication path between these devices. As a running program, there may be an operating system and various applications. Although not shown, it includes a power supply.

In the learning step, an artificial intelligence-based search model is trained in the memory 130 by using the learning data stored in the storage unit 120. In the inference step, the unsupervised learning-based search model 210 and the artificial intelligence-based search model 220 stored in the storage unit 120 are executed in the memory 130. The corpus is stored in the storage unit 120 and a search method is performed based on a query input through the input/output unit 140.

[Example]

Hereinafter, an embodiment in which the method for providing search results according to the present invention is applied to patent documents owned by the Korea Institute of Industrial Technology will be described.

Artificial intelligence-based search model

As an artificial intelligence-based search model, the open source CEDR-KNRM model was used. In brief, the CEDR-KNRM model is a structure that processes BERT and KNRM in parallel, and details can be checked from related papers and published source codes. According to Paper With Code, it was confirmed that the model had the best performance at the time of implementing the embodiment, and there is no change until the filing date.

Corpus

The number of patent documents to be searched, that is, document bundles, was approximately 3,000, and since the length of all documents exceeded 512 tokens that BERT could process, a corpus was formed by dividing each paragraph. The corpus contains approximately 120,000 passages.

The relationship between each passage and the extracted patent document is stored so that it can be referred to at a later stage of inference.

Create pseudo-query

At least one keyword or phrase was extracted as a pseudo-query by applying the keyword extraction technique to the passage formed by each paragraph of the patent document. The technique used here is RAKE, and the paper and open source are open to the public.

If the same pseudo-query was redundantly removed, approximately 40,000 pseudo-queries were generated.

Query-document relationship formation

Each pseudo-query was input, and 300 passages were extracted from the corpus using BM25, a search model based on the previously known unsupervised learning methodology. Since there were approximately 40,000 pseudo-queries, the total number of extracted passages was 12,000,000. This is an insufficient amount of data to train the CEDR-KNRM model.

Pseudo-labeling

Among 300 passages extracted by each pseudo-query, the upper m passages were classified as positive learning data, and the lower p passages were classified as negative learning data. Since the CEDR-KNRM model also uses neutral training data, the remaining passages excluding the upper m passages and the lower p passages were classified as neutral training data.

Hyperparameters such as m and p were adjusted by verifying the performance of the search model after learning by the inventor reviewing the search results. Since it is a dataset without ground truth, some of the prepared training data can be used as verification data, but since it is a field that can be verified by hand by the inventor, the performance was verified manually.

Reasoning stage

When a user inputs a query composed of a plurality of keywords, 300 passages are first retrieved from the corpus using BM25. The retrieved 300 passages are then provided as a trained CEDR-KNRM model along with a query entered by the user. The CEDR-KNRM model reranks 300 passages. The order of the patent documents is arranged to correspond to the order of the passages by searching the patent documents corresponding to the 300 re-ranked messages. In the case where there is one belonging to the same patent document among the retrieved packages, the corresponding patent documents are sorted according to the order of the highest-valued passages, and the subsequent ones are ignored when sorting the patent documents. Therefore, search results are less than or equal to 300.

Compare search results

In FIGS. 5 and 6, the results searched by the search method developed according to the present embodiment and the results searched by other search methods are described in comparison.

5 is a search result among patents owned by the Korea Institute of Industrial Technology by entering a query including two keywords "wheelchair" and "driving aid", and the left is the search result by the search method developed according to this embodiment, and the right Is the result retrieved from KIPRIS. As shown, in the search method developed according to the present embodiment, documents including words that exactly match the keywords "wheelchair" and "driving report", as well as other devices that can be used by people with disabilities While patents are also searched, in KIPRIS, only documents containing words that exactly match the keywords "wheelchair" and "driving report" are searched.

6 is a search result among patents held by the Korea Institute of Industrial Technology by entering a query including six keywords such as "scanning sonar", "safety monitoring", "marine operation equipment", "marine monitoring", and "aquaculture monitoring" As, the upper side is the search result by the search method developed according to the present embodiment, and the lower side is the search result by Google Patent Search. However, in the case of the Google patent search, since the search was not performed when the number of keywords included in the query exceeded 5, the search was performed excluding the keyword "Farm Monitoring". As shown, in the search method developed according to the present embodiment, even if it does not exactly match the keyword included in the query, it searches for a document containing words that are semantically similar, whereas in the Google patent, there is no search result. These results are the same for KIPRIS (not shown).

As can be seen from FIGS. 5 and 6, the search method according to the present invention using an artificial intelligence-based search model does not include keywords included in the input query, but can additionally search documents that are semantically relevant. have.

[Processing method for sentence type query]

7 shows an example of processing a sentence type query according to the present invention. In general, search models based on unsupervised learning often input keyword queries rather than sentence-type queries. The search model based on artificial intelligence can process sentence-type queries as inputs, but it is not easy to prepare them when there is no learning data. For academic research, sentence-type query (or explanation)-passage pair data such as SQuAD and TREC Robust are provided as English data sets, and KorQuAD provides sentence-type query-passage pair data as a Korean data set. These data are from data such as Wikipedia. However, depending on the field, it may not be enough to just learn it with data for academic research purposes, and to learn an artificial intelligence-based search model, not only positive data but also negative data are required. There is a limit to providing.

Various techniques can be proposed as a method for solving this problem, but in the present embodiment, a method of converting a sentence-type query into a keyword query by sequential labeling is selected. Sequence labeling is a process of tagging sentences with POS, performing morpheme analysis, removing elements that have a low degree of contributing to search, such as investigation and ending, and leaving only nouns and noun roots. In addition, when converting a sentence-type query into a keyword query through sequence labeling, the part-of-speech and/or morpheme to be left may vary depending on the field to which the document to be searched belongs. In some cases, if it is judged as stop words, it can be removed even as a noun. The determination of whether or not a stopword can be determined includes a method such as querying a stopword dictionary created in advance.

The converted keyword query is then provided as input to the unsupervised learning-based search model (BM25) and the artificial intelligence-based search model in the manner described above. Depending on the length of the sentence, the number of keywords may be too small or too large, but as shown in FIGS. 5 and 6, the artificial intelligence-based search model according to the present invention is determined to be related in terms of content other than the input keyword query. Passages can be provided as search results, and even if there are too many input keywords, related passages can be provided as search results, so the number of keywords included in the keyword query is less affected. In development, as a hyper parameter, the minimum and maximum number of keywords included in the keyword query may be limited.

The above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (9)

A computer-implemented method for providing a search result corresponding to a sentence type query from a corpus including a plurality of passages,
(a) converting a sentence-type query into a keyword query through sequence labeling;
(b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology;
(c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And,
(d) the step of outputting a list of search results corresponding to the passages re-ranked in step (c)
Including,
Each passage of the corpus is a part of each document of the document bundle consisting of a plurality of documents to be searched,
The artificial intelligence-based search model learned by the Wicksupervision methodology is used in step (a) by using the plurality of passages included in the corpus and a pseudo-query generated from each passage. Generated by a search model based on unsupervised learning that includes the search model
Computer-implemented method for retrieving documents.
The method according to claim 1,
In the step (a), a lower limit value and an upper limit value of the number of keywords included in the keyword query are preset
Computer-implemented method for retrieving documents, characterized in that.
The method according to claim 1,
In the step (a), stop words included in the sentence-type query are removed.
Computer-implemented method for retrieving documents, characterized in that.
The method of claim 3,
Whether or not it corresponds to the stopword is determined by inquiring the stopword dictionary prepared in advance.
Computer-implemented method for retrieving documents, characterized in that.
An apparatus for providing a search result corresponding to a sentence-type query from a corpus including a plurality of passages,
At least one processor; And
Including at least one memory for storing computer-executable instructions,
The computer-executable instruction stored in the at least one memory, by the at least one processor,
(a) converting a sentence-type query into a keyword query through sequence labeling;
(b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology;
(c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And,
(d) the step of outputting a list of search results corresponding to the passages re-ranked in step (c)
To run, but
Each passage of the corpus is a part of each document of the document bundle consisting of a plurality of documents to be searched,
The artificial intelligence-based search model learned by the Wicksupervision methodology is used in step (a) by using the plurality of passages included in the corpus and a pseudo-query generated from each passage. Learning based on a pseudo-label generated by a search model based on unsupervised learning including a search model
Device for retrieving documents.
The method of claim 5,
In the step (a), a lower limit value and an upper limit value of the number of keywords included in the keyword query are preset
Device for searching for a document, characterized in that.
The method of claim 5,
In the step (a), stop words included in the sentence-type query are removed.
Device for searching for a document, characterized in that.
The method of claim 7,
Whether or not it corresponds to the stopword is determined by inquiring the stopword dictionary prepared in advance.
Computer-implemented device for retrieving documents, characterized in that.
A computer program for providing a search result corresponding to a sentence-type query from a corpus including a plurality of passages,
It is stored in a non-transitory storage medium, and by the processor,
(a) converting a sentence-type query into a keyword query through sequence labeling;
(b) retrieving N passages corresponding to the keyword query from the corpus by a search model based on an unsupervised methodology;
(c) Re-ranking the N passages retrieved in step (b) based on the keyword query by the artificial intelligence-based search model learned by the weak-supervision methodology. step; And,
(d) the step of outputting a list of search results corresponding to the passages re-ranked in step (c)
Contains a command that causes it to be executed,
Each passage of the corpus is a part of each document of the document bundle consisting of a plurality of documents to be searched,
The artificial intelligence-based search model learned by the Wicksupervision methodology is used in step (a) by using the plurality of passages included in the corpus and a pseudo-query generated from each passage. Learning based on a pseudo-label generated by a search model based on unsupervised learning including a search model
A computer program stored on a non-transitory storage medium for retrieving documents.

Images