KR20220117944A - Method for batch-processing of data preprocessing and training process for information retreival model - Google Patents

Abstract

Provided is a method for batch-processing of data preprocessing and training procedures for an artificial intelligence-based information search model. A batch-processing procedure comprises the following procedures of: when a document for an information search model using a passage unit as a search target is given, dividing the document into passages to generate corpuses; and training the information search model based on the Weaksupervision methodology by using at least some of the passages in the corpuses. Since the method can batch-process the preprocessing procedure of dividing the document into passages having a length processable by the artificial intelligence-based information search model to generate corpuses when the document is given, the method can reduce costs and time required for a preprocessing procedure. Moreover, since the method can batch-process the procedure of training the information search model based on the Weaksupervision methodology by using at least some of the corpuses generated in the preprocessing procedure, the method can reduce costs consumed for the training of the artificial intelligence-based information search model.

Description

Method for batch-processing of data preprocessing and training process for information retreival model

The present invention relates to a method for collectively processing data pre-processing and learning processes for an IR model, and more specifically, for an IR model that uses a passage unit as a search target, when a document is given, it is divided into passages to form a corpus It relates to a method of collectively processing the process of making an IR model based on the Wick Supervision methodology using at least some passages in the generated corpus.

Search technology has been continuously developed since Google introduced the PageRank search technology based on graph theory. This search technology is based on unsupervised learning, and it was possible to search only if a bundle of documents was given. A representative example of a search model based on unsupervised learning is BM25, and it shows very improved performance when used together with the query expansion technique called RM3. As an open source, Anserini is widely used in academic fields and fields.

On the other hand, various search models have been proposed according to research in the academic field to apply artificial intelligence techniques in the field of natural language processing. BERT, announced by Google in 2018, showed good performance in several natural language processing fields, and since then, research to use various language models for information retrieval has continued. According to a researcher at the University of Waterloo, Canada, named Jimmy Lin, the IR models of deep learning before BERT had similar or inferior performance to Anserini, an IR model based on unsupervised learning methodology, but after BERT, The proposed IR models are said to have improved performance over Anserini (refer to Lin, Jimmy. "The Neural Hype, Justified! A Recantation.").

However, the AI-based information retrieval model has some limitations. First, IR models based on unsupervised learning generally do not have a problem even if the length of a document is long, but AI-based IR models have a limitation in the length of documents that can be processed most of the time. For example, in the case of BERT, the maximum number of tokens that can be processed is limited to 512. Therefore, there is no problem when a corpus composed of short texts is used as a search target, but it is difficult to apply when searching for a long document such as a patent or a thesis. As another problem, in order to use an AI-based search model for inference, it must first be trained, and this training requires a large amount of labeled data. Labeled data must be processed and provided by humans, but considering the amount of data required for learning, the cost of labeling is too high, which is uneconomical.

In order to solve the first problem mentioned above, when an AI-based IR model is used, a method of dividing a document into a small size that the IR model can process was proposed as the basic unit of the search target. . However, there is a problem in that the pre-processing process of dividing the document into passages and making them a search target takes a lot of time.

As one of the methods to solve the second problem mentioned above, the Wick Supervision methodology has been proposed. For example, in Korean Patent No. 10-2197645, filed and registered by the present applicant, keywords are extracted from each passage, which is a search target, using a keyword extraction algorithm, and this is used as a pseudo-query, and unsupervised learning A method for generating a pseudo-label between a pseudo-query and a passage using an IR-based IR model and training an AI-based IR model using this weak label is disclosed. However, there is a problem in that such a learning process is done manually, and thus a lot of cost is required.

[1] https://paperswithcode.com/task/ad-hoc-information-retrieval [2] MacAvaney, Sean, et al. "CEDR: Contextualized embeddings for document ranking." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019. [3] Dai, Zhuyun, and Jamie Callan. "Deeper text understanding for IR with contextual neural language modeling." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.

The present invention is to solve the above problem, and when a document is given, it is divided into passages having a length that an AI-based IR model can process, and a pre-processing process to make a corpus and at least some passages in the generated corpus It is to provide a method for collectively processing the process of learning the IR model based on the Wick Supervision methodology using

In order to solve the above problems, according to an aspect of the present invention, a computer-implemented method for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus, (a) receiving a parameter necessary for batch processing, which is predetermined based on information on a plurality of documents to be searched; (b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a); (c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b); (d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c); (e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); and, (f) using the pseudo-label generated in step (e), to train an artificial intelligence-based IR model. A method of implementation is provided.

According to another aspect of the present invention, there is provided an apparatus for collectively processing generation of a corpus for an IR model and training of an IR model based on the corpus, comprising: at least one processor; and at least one memory for storing computer-executable instructions, wherein the computer-executable instructions stored in the at least one memory are executed by the at least one processor (a) for a plurality of documents to be searched. receiving a parameter necessary for batch processing, which is predetermined based on the information; (b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a); (c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b); (d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c); (e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); and, (f) a device for batch processing data pre-processing and learning process for an IR model so that the step of training the AI-based IR model is executed using the pseudo-label generated in step (e). is provided

According to another aspect of the present invention, a computer program for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus, stored in a non-transitory storage medium, by a processor ( a) receiving a parameter necessary for batch processing, which is predetermined based on information on a plurality of documents to be searched; (b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a); (c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b); (d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c); (e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); And, (f) using the pseudo-label generated in step (e), the step of learning the AI-based IR model is executed, including a command to be executed, data pre-processing and learning process for IR model A computer program stored in a non-transitory storage medium for batch processing is provided.

According to the present invention, when a document is given, the preprocessing process of making a corpus by dividing it into passages having a length that can be processed by the AI-based IR model can be processed in a batch, so the time and cost required for the preprocessing process can save

In addition, according to the present invention, since the process of learning the IR model based on the Wick Supervision methodology using at least some passages in the corpus generated in the pre-processing process can be processed collectively, the AI-based IR model can reduce the cost of learning.

1 is a flowchart illustrating a method for learning a conventional artificial intelligence-based information retrieval model.
2 is a flowchart illustrating an information retrieval method using a conventional artificial intelligence-based information retrieval model.
3 is a diagram schematically illustrating a conventional information retrieval model.
4 is a diagram schematically showing an apparatus for performing an information retrieval method using an artificial intelligence-based information retrieval model.
5 is a flowchart illustrating a method for collectively processing data pre-processing and a learning process for an information retrieval model according to an embodiment of the present invention.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings. The same or similar reference numerals are assigned to the same or similar components, and overlapping descriptions thereof are omitted. In describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. The accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It should be understood to include what is or what is replaced.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various components, but these terms are used only for the purpose of distinguishing one component from other components, and the corresponding components are defined by these terms. not limited by The singular expression includes the plural expression unless the context clearly dictates otherwise.

As used herein, terms such as “comprises”, “comprises” or “have” are to be understood as limiting the existence of a feature, step, element, or combination thereof described in the specification, and one or more other It is not intended to exclude the possibility that features, steps, components, or combinations thereof may be present or added.

Hereinafter, in order to help the understanding of the present invention, a method of learning an artificial intelligence-based information search model disclosed in Korean Patent No. 10-2197645 filed and registered by the applicant of the present invention and a method of providing search results using the method First, after a brief description with reference to FIGS. 1 to 4 , a batch processing method of data pre-processing and a learning process according to the present invention will be described.

1 shows a method for learning an artificial intelligence-based information retrieval model according to the present invention, and FIG. 2 shows a step of performing inference using the artificial intelligence-based information retrieval model according to the present invention, 3 schematically shows an example of an information retrieval model according to the present invention. Hereinafter, the present invention will be described with reference to these drawings.

[Learning stage of information retrieval model based on artificial intelligence]

information retrieval model

In this specification, the IR model is divided into a 'search model based on an unsupervised methodology' and an 'artificial intelligence-based search model'. The former means information retrieval models based on statistical or other unsupervised learning methodologies such as BM25 and QL (Query Likelihood). It means an information retrieval model prepared by learning, including

In the present invention, the former search model based on the unsupervised learning methodology is used to prepare pseudo-labels in the learning stage, and is used to retrieve passages from the corpus in the inference stage, and the latter is used to It is used to re-rank the passage retrieved in the reasoning step.

Information retrieval models for which open sources exist are disclosed in the Ad-Hoc Information Retrieval section of Paper With Code.

Search model based on unsupervised learning methodology

As search models based on unsupervised methodologies, BM25 and Query Likelihood (QL) are known, and Anserini is widely used as an open source. Anserini is a search model that uses RM3, a query expansion methodology, along with BM25. According to the research results, the performance of Anserini is similar to that of deep learning-based search models such as DRMM, KNRM, and PACRR, but it is known that the performance of Anserini is inferior to that of language model-type search models such as BERT.

The search model based on the unsupervised learning methodology does not require learning and determines the similarity between the query and the document based on statistical theories.

AI-based search model

As AI-based search models, deep learning-based search models such as DRMM, KNRM, and PACRR, and transformer or language model-based search models such as BERT are known. According to Paper With Code, the best performing SOTA (State-of-the-Art) search model as of the filing date of this patent is CEDR, which has a structure that combines the language model-type BERT and the deep-learning-type search model. is understood to be

The AI-based search model is completed by learning the similarity between the query and the document based on the training data. Academically, AI-based search models are trained using datasets that provide query-document relationships, such as TREC, SQuAD, and MS Marco.

The search engine of the language model series such as BERT uses the self-supervision learning method in the pre-training stage, but fine-tuning to learn it suitable for the information retrieval task. In the -tuning) stage, supervised learning is required based on a dataset on the query-document relationship, similar to the deep learning-based search model.

Weaksupervision Methodology

Query-document relation datasets such as TREC, SQuAD, and MS Marco that are used academically correspond to ground truth extracted based on real users' search activity, or some use the title of a document as a query. However, in the actual field, only a document corpus exists, and there is no query input by the user to search for it, or even if there is, it is very insufficient to train an AI-based search model. In the present invention, when only a document bundle exists, a pseudo-query is generated from each document, a pseudo-query-document relationship is formed using the generated pseudo-query, and a pseudo-label is generated. , a methodology for learning an AI-based search model using this is provided. The Wick Supervision methodology for the AI-based search model according to the present invention broadly includes the following steps: 1) generating a pseudo-query from each document in the document bundle, 2) the generated pseudo-query Creating a pseudo-query-document relationship using . Hereinafter, each of the above-described steps will be described by dividing the items.

1) Steps to create a pseudo-query

One or more keywords are extracted from each document in the document bundle and determined by pseudo-query. A method of extracting a keyword from a document uses a known keyword extraction technique. Keyword extraction algorithms can also be largely divided into techniques based on unsupervised learning and techniques based on supervised learning, and open sources for a number of techniques exist.

Although most techniques are methods of extracting multiple keywords or phrases from a document, recently, as a method based on supervised learning, a technique called Doc2Query, which uses BERT to generate a query in the form of a natural language sentence, has also been released as an open source. there is a bar

2) Create a pseudo-label

M documents are retrieved from the document bundle by using the pseudo-query extracted from each document as an input to a search model based on unsupervised learning methodologies such as BM25. In this case, the document from which the corresponding pseudo-query is extracted is highly likely to be included in the top among the M documents, but may not be included.

Among the retrieved M documents, the top m (<M) documents are labeled as positive training data, and at least some of the remaining documents among the M documents are labeled as negative training data. According to the AI-based search model, it is possible to additionally label other parts as neutral training data. In general, positive data and negative data are necessary, but neutral data is not necessarily required.

Here, the number of retrieved documents M, the number of positive learning data m, the number of negative learning data, and the number of neutral learning data are integers, and may be determined differently depending on the characteristics of the document bundle as a kind of hyperparameter. Developers can increase the accuracy of AI-based search models by adjusting these hyperparameters in an experimental or theoretical approach according to the characteristics of the document bundle.

3) Learning the AI-based search model

Since a query-document relationship is required to train an AI-based search model, the training data is provided in the form of a relationship between a query used for retrieval and a document retrieved by the query. For example, (pseudo-query, positive document) may be provided in the form of a pair of two data, or (pseudo-query, positive document, negative document) may be provided in a form in which three pieces of data are related. Of course, up to a neutral document can also be associated with four data.

The amount of data required for learning is not known precisely, and is mainly confirmed by experimental methods. However, the amount of data required for training increases according to the number of parameters included in the model. For search models based on the BERT series of language models, pre-trained models are generally published based on Wikipedia, etc. Therefore, a small amount of data is required.

In academic research, the amount of learning data is determined according to the available dataset, but in the actual field, the amount of learning data may be determined according to the amount of documents included in the document bundle. When the amount of documents included in the document bundle is insufficient for learning, other documents in a similar field may be added to the document bundle, or the amount of data may be increased by using a data augmentation technique.

When training an AI-based model, it is generally desirable to verify performance by dividing the data into training data and verification data. For example, 80% of the total data may be used for learning, and the remaining 20% of the data may be used for verification.

[Inference stage]

In the present invention, the reasoning step refers to a process of providing a search result in response to a query input by a user. The reasoning step can be roughly divided into the following steps: a retrieving step, a re-ranking step, and an output step. Hereinafter, each step will be described in detail.

retrieval stage

It is a step of extracting N documents from a document bundle which is a search target by a search model based on an unsupervised learning methodology. In general, the AI-based search model takes considerably longer for inference than the search model based on the unsupervised learning methodology. Accordingly, when the AI-based search model is applied to all documents included in the document bundle, it takes too long, and the user's convenience deteriorates. Therefore, after retrieving a large number of documents by the search model based on the unsupervised learning methodology, which is relatively fast in the first place, it is common to apply the AI-based search model only to the secondly retrieved documents. Research to omit the retrieval step by improving the inference speed of the AI-based search model is continuously being conducted, but it is understood that the inference speed has not yet reached a sufficiently fast inference speed considering the user convenience.

At this stage, accuracy is important, but recall is more important. On the other hand, in the second re-ranking stage, accuracy is more important than recall. One of the methods to increase the recall is to retrieve a number of documents several times larger than the number of documents to be provided in the search results. Another method is to increase the recall rate of the search model based on the unsupervised learning methodology. By using a technique called DeepCT Index using BERT, it is possible to increase recall while using a search model based on the same unsupervised learning methodology. In order to retrieve a large amount of documents, the time required for retrieval increases in proportion to the number of documents. In the case of using the DeepCT Index technique, it is possible to reduce the search time by reducing the number of documents to be retrieved, while achieving the same level of recall as when retrieving a number of documents several times larger than that.

As a search model based on the unsupervised learning methodology, the same search model used in the learning step may be used, or a different search model may be used. It is important to select a search model with high recall in the relevant field. In academic research, Anserini (BM25+RM3) is widely used as an open source-based search model.

re-ranking stage

The relevance between the retrieved N documents and the query entered by the user is evaluated by a search model based on the unsupervised learning methodology and sorted according to the order of relevance. In this step, the relation between the retrieved N documents and the query entered by the user is re-evaluated by the AI-based search model and then rearranged, that is, re-ranked. This step is performed by the AI-based search model learned in the above-described learning step.

Search result output

The N documents reranked by the AI-based search model are sorted according to the order of relevance and output as a search result.

[Handling in special cases]

Matters related to the length of the document

In general, search models based on unsupervised learning methodologies have no problem in execution even if the length of documents is long, but AI-based search models often have a limit on the maximum length of documents that can be processed. In particular, AI-based search models introduced after BERT have a limited number of maximum tokens that can be processed. For example, in the case of BERT, the maximum number of tokens that can be processed is limited to 512.

When there is a limit on the maximum length of a document that can be processed by the AI-based search model, each document in a document corpus can be divided into passages with a length less than the limited length and can be searched. In the present specification, a bundle of passages composed of short texts is referred to as a 'corpus' in order to distinguish it from a document bundle composed of long documents.

In the learning stage, the passages included in the corpus are the same as described above except that the passages are replaced with documents. That is, after extracting a pseudo-query from a passage and generating a pseudo-label using a search model based on an unsupervised learning methodology, an AI-based search model is trained. The AI-based search model learns based on the relationship between the pseudo-query and the passage, not the relationship between the pseudo-query and the entire document.

In the reasoning step, it is necessary to divide each document in the document bundle into passages to make a corpus, and to perform preprocessing of storing the correspondence between passages and documents in a form that can be referred to. In the retrieval step, N passages are retrieved from the corpus based on the query input by the user, and in the reranking step, the retrieved N passages are reranked.

Documents that are not passages in the search results should be provided in order of relevance. To this end, with reference to the correspondence between the passages and documents provided in the pre-processing step, the documents are arranged so as to correspond to the re-ranked passage's relevance sort order and provided as a search result. Since several passages are separated from one document, a plurality of passages extracted from one document may be included in the reranked result. In this case, the sort order of documents may correspond to the order of passages with the highest degree of relevance.

document expansion

In the case of a long document, it contains multiple paragraphs, and the content of each paragraph does not necessarily match the topic. Accordingly, when each paragraph is divided into passages and stored as a corpus, there is a possibility that the relevance is evaluated low even though the paragraphs are separated from related documents.

In order to prevent such a problem, the title of the document or a phrase equivalent thereto may be added to each passage separated from the document. That is, the same phrase is added to each passage separated from one document. In this specification, the extension by adding a phrase having relevance to the passage is expressed as a key word 'document extension'. A phrase corresponding to the title of the document may be generated using, for example, a text summarization technique. The document summarization technique is largely divided into extractive summarization and abstract summarization, both of which can be applied. The technique for generating additional phrases for document extension is not necessarily limited to the document summary technique, and any technique that can compressively express the subject of the document may be used.

For example, a plurality of keywords are included in a query input by a user. Some keywords may be included in the original message, but other keywords may be included in a phrase added by document expansion. Since the phrase added by the document extension includes the subject of the entire document, there is a possibility that the corresponding passage contains the content intended by the user to search. If the corresponding phrase was not added by document extension, a passage that would not have been retrieved may be retrieved by document extension. This may increase the recall of the retrieval step. As described above, it is important that the recall is high in the retrieval step.

If the length of the phrase added to the passage due to the document extension is too long, the length of the passage increased by the document extension may exceed the length that the AI-based search model can handle. In order to minimize the side effects caused by this, it is preferable that the text added by the document extension be placed at the front of the passage. Even if the rear end of a passage whose length is increased due to document expansion is cut off, if the keyword included in the query entered by the user is not included in the rear end, the passage will still be extracted in the retrieval stage.

domain adaptiveness

When an AI-based search model is trained as a document of a specific domain, it can be used as a search model specialized for that domain. For example, when fine-tuning a BERT that has been pre-trained with a general document bundle such as Wikipedia as a document of a specific domain, in the pre-learning stage, general relationships between vocabularies are learned, and in fine-tuning, a vocabulary specific to that domain is learned. will learn

A search model specialized for a specific domain may have relatively poor performance for other domains, but it improves performance in that domain.

4 shows a computer device for performing a search method according to the present invention.

Since the search method and the learning method according to the present invention have been described in detail with reference to FIGS. 1 to 3 , an apparatus 100 for performing such a search method will be briefly described with reference to FIG. 4 .

Referring to FIG. 4 , the computer device 100 includes a processor 110 , a non-volatile storage unit 120 for storing programs and data, a volatile memory 130 for storing programs being executed, and information between a user and a user. It consists of an input/output unit 140 for inputting and outputting , and a bus, which is an internal communication path between these devices. The running program may include an operating system and various applications. Although not shown, it includes a power supply unit.

In the learning step, an AI-based search model is learned in the memory 130 using the learning data stored in the storage unit 120 . In the reasoning step, the memory 130 executes the unsupervised learning-based search model 210 and the artificial intelligence-based search model 220 stored in the storage unit 120 . The corpus is stored in the storage unit 120 , and a search method is performed based on a query input through the input/output unit 140 .

Hereinafter, a method for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus will be described with reference to FIG. 5 , as an embodiment according to the present invention.

Step of receiving parameters required for batch processing (S10)

Parameters required for each process are input so that the following processes can be batch-processed. These parameters may be input in advance, or input may be requested from the user before each process starts. When input is received in advance, from the data pre-processing process to the learning process of the information retrieval model, all processes can be batch-processed.

Corpus creation process (S20)

Given documents to be searched, each document is divided into passages having a predetermined size or according to a predetermined rule.

The predetermined size is determined in consideration of the text size that the AI-based IR model to be applied can process. For example, in the case of BERT, the maximum number of tokens that can be processed is 512, so the size of the text is determined by excluding the number of tokens to be used for queries and special tokens. For example, if you limit the number of tokens for a query to 11, that is, roughly limit the number of keywords included in a query to 11, a total of 12 tokens, including the special token [CLS], should be excluded. do. In that case, the maximum length of passages to be included in the corpus is limited to 500 tokens.

As a predetermined rule, for example, a document may be divided in units of paragraphs. Since each paragraph of the document is separated by a return sign, which is a typesetting mark, the document can be divided into paragraphs based on the return sign.

On the other hand, when the document is divided into a predetermined size, there is a possibility that the content to be searched is divided into different passages. In order to prevent this problem, when dividing a document into passages having a certain length, they are partially overlapped. For example, if 50% overlap, the subsequent passage contains 50% of the last portion of the previous passage. The overlapping ratio is not necessarily set to 50%, and the user can arbitrarily designate it in consideration of the characteristics of the corpus. However, in order for the corpus generation process to be batch-processed, it is preferable for the user to initially determine and input the overlapping ratio of passages.

Creating document-passage relationship data

In order to provide a later search result in a document unit rather than a passage unit, a relationship between a document and a passage extracted therefrom is created and stored. This process may be processed in batch following the previous processes.

Create indexing file for information retrieval model based on unsupervised learning

The information retrieval model based on unsupervised learning uses a pre-generated indexing file to improve the search speed. For the information retrieval model based on unsupervised learning used in the retrieval step, an indexing file is generated from passages included in the corpus. This process can be processed in batches following the previous processes.

Pseudo-query generation process (S30)

After dividing each document into a plurality of passages, one or more keywords or phrases are extracted as pseudo-queries by applying a predetermined keyword extraction technique to each of these passages. It is preferable that the keyword extraction technique to be applied is predetermined so that the present process can be batch-processed following the previously performed process.

Query-document relationship formation process (S40)

With each extracted pseudo-query as input, a predetermined number of passages are extracted from the corpus using a search model based on a previously known unsupervised learning methodology. Similarly, for batch processing, it is preferable that the number of passages to be extracted from the corpus is predetermined and input.

Pseudo-Labeling Process (S50)

Among the passages extracted by each pseudo-query, the upper m passages were classified as positive learning data, and the lower p passages were classified as negative learning data. If necessary for learning, the remaining passages except for the upper m passages and the lower p passages may be classified as neutral learning data.

Hyperparameters such as m and p are generally adjusted after learning while examining the performance of the IR model, but it is preferable to be determined in advance so that they can be processed in batches following the previous processes.

Learning process (S60)

An artificial intelligence-based information retrieval model is trained using the generated pseudo-label. This process can be batch-processed following the previous processes given a pseudo-label. However, it is preferable that the criterion for stopping the learning process is predetermined. For example, when a loss in the learning process becomes smaller than a predetermined value, the learning process may be stopped.

reasoning phase

After the data pre-processing and learning process for the AI-based IR model are batch-processed through the above process, the user can perform a search operation using the completed IR model.

When a user inputs a query composed of a plurality of keywords, a predetermined number of passages are retrieved from the corpus using BM25. The retrieved passage is then provided as a trained IR model along with the query input by the user. The IR model reranks a given passage. By inquiring the documents corresponding to the reranked passages, the order of the documents is arranged to correspond to the order of the passages.

The foregoing detailed description should not be construed as restrictive in any way but as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (9)

A computer-implemented method for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus,
(a) receiving a parameter necessary for batch processing, which is predetermined based on information on a plurality of documents to be searched;
(b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a);
(c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b);
(d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c);
(e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); and,
(f) using the pseudo-label generated in step (e) to train an artificial intelligence-based information retrieval model
A computer-implemented method for batch processing data pre-processing and learning process for an information retrieval model, including.
The method according to claim 1,
The parameters required for the batch processing include a parameter related to a method for classifying documents in step (a), a parameter related to a keyword extracted in step (c), and a parameter related to the number of passages to be extracted in step (d) and hyperparameters for generating the pseudo-label in step (e).
A computer-implemented method for batch processing data pre-processing and learning process for an information retrieval model, characterized in that
3. The method according to claim 2,
The parameter regarding the method of classifying the document includes information on the size of passages in which the documents are to be divided and the overlapping ratio of each passage
A computer-implemented method for batch processing data pre-processing and learning process for an information retrieval model, characterized in that
The method according to claim 1,
Storing the relationship between the document and the passages delimited therefrom.
A computer-implemented method for batch processing data pre-processing and learning process for an information retrieval model, characterized in that it further comprises.
An apparatus for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus,
at least one processor; and
at least one memory for storing computer-executable instructions;
The computer-executable instructions stored in the at least one memory are executed by the at least one processor,
(a) receiving a parameter necessary for batch processing, which is predetermined based on information on a plurality of documents to be searched;
(b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a);
(c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b);
(d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c);
(e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); and,
(f) using the pseudo-label generated in step (e) to train an artificial intelligence-based information retrieval model
A device that batch-processes data pre-processing and learning processes for an IR model to be executed.
6. The method of claim 5,
The parameters required for the batch processing include a parameter related to a method for classifying documents in step (a), a parameter related to a keyword extracted in step (c), and a parameter related to the number of passages to be extracted in step (d) and hyperparameters for generating the pseudo-label in step (e).
A device for batch processing data pre-processing and learning process for an information retrieval model, characterized in that
7. The method of claim 6,
The parameter regarding the method of classifying the document includes information on the size of passages in which the documents are to be divided and the overlapping ratio of each passage
A device for batch processing data pre-processing and learning process for an information retrieval model, characterized in that
6. The method of claim 5,
Storing the relationship between the document and the passages delimited therefrom.
Device for batch processing data pre-processing and learning process for the information retrieval model, characterized in that it further comprises.
A computer program for collectively processing generation of a corpus for an IR model and learning of an IR model based on the corpus,
It is stored in a non-transitory storage medium, and by the processor,
(a) receiving a parameter necessary for batch processing, which is predetermined based on information on a plurality of documents to be searched;
(b) generating a corpus by dividing each of the plurality of documents into passages based on the parameters input in step (a);
(c) extracting a keyword to be used as a pseudo-query from each of at least some passages included in the corpus generated in step (b);
(d) extracting a specified number of passages from the parameters input in step (a) using the pseudo-query and unsupervised learning-based IR model extracted in step (c);
(e) generating a pseudo-label based on the hyper parameter specified in the parameter input in the step (a); and,
(f) using the pseudo-label generated in step (e) to train an artificial intelligence-based information retrieval model
A computer program stored in a non-transitory storage medium for batch processing of data pre-processing and learning process for an information retrieval model, including instructions to be executed.

Images