KR102216065B1 - Method for providing search result for video segment - Google Patents

Abstract

The present invention provides a search method of a video segment or an image using an artificial intelligence-based search model. A video segment or an image does not basically include text-based information, but a corresponding passage can be generated by using a technique such as object recognition and caption generation. If summary information on a video or an image exists, the information can also be included in the passage. A video segment or an image can be searched by using the generated text passage. If an artificial intelligence-based search model is used, search results with improved performance can be obtained in comparison to using only an unsupervised search model.

Description

Method for providing search result for video segment}

The present invention relates to a method of providing a search result for a video segment, and more specifically, to a method of providing a search result for a video segment using an information retrieval model learned based on a wick supervision methodology. The present invention can also be used for image search.

Search technology has been continuously developed since Google introduced the PageRank search technology based on graph theory. This search technique was based on unsupervised learning, and it was possible to search only given a bunch of documents. BM25 is a representative search model based on unsupervised learning, and it shows very improved performance when used with a query expansion technique called RM3. As an open source, Anserini is widely used in the academic field and in the field.

Meanwhile, in the field of natural language processing, various search models have been proposed in accordance with research in the academic field to apply artificial intelligence techniques. For example, deep learning-based search models such as DRMM, KNRM, and PACRR have been proposed. BERT, released by Google in 2018, has shown good performance in various natural language processing fields, and research has been conducted to use it as a search model based on a transformer or language model.

In the Ad-Hoc Information Retrieval section of Paper With Code, a website that introduces artificial intelligence models that are open source in each field, the current point of view of artificial intelligence-based search models including Anserini, a search model based on unsupervised learning. SOTA (State-of-the-Art), that is, the search model that shows the best performance can be identified. According to a researcher at the University of Waterloo, Canada, named Jimmy Lin, deep learning search models before BERT, i.e., DRMM, KNRM, and PACRR, are similar to Anserini, a search model based on unsupervised learning methodology. Although the performance is similar or worse, the models proposed after BERT are said to have improved performance over Anserini (see Lin, Jimmy. "The Neural Hype, Justified! A Recantation."). Such matters can also be checked on the leader board of the Ad-Hoc Information Retrieval item of the above Paper With Code. From these academic research results, it can be seen that the accuracy of the search results can be improved by the search model based on artificial intelligence.

However, artificial intelligence-based search models have some limitations.

In order to use an artificial intelligence-based search model for inference, it must first be trained, and such learning requires a large amount of labeled data. Labeled data must be basically processed and provided by humans, but it is uneconomical because the cost for labeling is too high when considering the amount of data required for learning.

As another problem, search models based on unsupervised learning generally do not matter even if the length of the document is long, but most of the search models based on artificial intelligence have limitations in the length of documents that can be processed. In the case of BERT, for example, the maximum number of tokens that can be processed is limited to 512. Therefore, there is no problem in the case of searching for a corpus consisting of short articles, but there is difficulty in applying it in the case of searching for long documents such as patents and papers.

On the other hand, in the case of moving pictures or images, it is difficult to search using an information retrieval technique because text information is not basically included.

[1] https://paperswithcode.com/task/ad-hoc-information-retrieval [2] MacAvaney, Sean, et al. "CEDR: Contextualized embeddings for document ranking." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019. [3] Dai, Zhuyun, and Jamie Callan. "Deeper text understanding for IR with contextual neural language modeling." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.

The present invention is to solve the above-described problem, and provides a method of providing a search result for a video segment or image by using a text-type query.

According to an aspect of the present invention, a computer-implemented method for providing a search result corresponding to a query input from a corpus including a plurality of passages, (a) unsupervised learning Retrieving N passages corresponding to the input query from the corpus by a search model based on methodology; (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages re-ranked in step (b), wherein the passage includes at least one of object information and subtitle information of a video segment. A computer-implemented method for retrieving segments is provided.

According to another aspect of the present invention, an apparatus for providing a search result corresponding to a query input from a corpus including a plurality of passages, comprising: at least one processor; And at least one memory storing a computer-executable instruction, wherein the computer-executable instruction stored in the at least one memory is provided by the at least one processor, (a) an unsupervised methodology Retrieving N passages corresponding to the input query from the corpus by a search model based on (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages re-ranked in step (b), wherein the passage includes at least one of object information and subtitle information of a video segment. An apparatus for searching for segments is provided.

According to another aspect of the present invention, as a computer program for providing a search result corresponding to a query input from a corpus including a plurality of passages, stored in a non-transitory storage medium, and by a processor, (a) unsupervised Retrieving N passages corresponding to the input query from the corpus by a search model based on an unsupervised methodology; (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages re-ranked in step (b), wherein the passage includes at least one of object information and subtitle information of a video segment. A computer program is provided that is stored on a non-transitory storage medium for retrieving segments.

According to another aspect of the present invention, a computer-implemented method for providing a search result corresponding to a query input from a corpus including a plurality of passages, comprising: (a) unsupervised learning ) Retrieving N passages corresponding to the input query from the corpus by a methodology-based search model; (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages reranked in step (b), wherein the passage includes an image including at least one of object information and caption information of the image. A computer-implemented method for searching is provided.

According to another aspect of the present invention, an apparatus for providing a search result corresponding to a query input from a corpus including a plurality of passages, comprising: at least one processor; And at least one memory storing a computer-executable instruction, wherein the computer-executable instruction stored in the at least one memory is provided by the at least one processor, (a) an unsupervised methodology Retrieving N passages corresponding to the input query from the corpus by a search model based on (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages reranked in step (b), wherein the passage includes an image including at least one of object information and caption information of the image. An apparatus for searching is provided.

According to another aspect of the present invention, as a computer program for providing a search result corresponding to a query input from a corpus including a plurality of passages, stored in a non-transitory storage medium, and by a processor, (a) unsupervised Retrieving N passages corresponding to the input query from the corpus by a search model based on an unsupervised methodology; (b) Re-ranking the N passages retrieved in step (a) based on the input query by an artificial intelligence-based search model learned by the weak-supervision methodology Step of becoming; And, (c) outputting a list of search results corresponding to the N passages re-ranked in step (b), wherein the passage includes an image including at least one of object information and subtitle information of the image. A computer program is provided that is stored on a non-transitory storage medium for retrieval.

According to the present invention, there is provided a method of providing a search result for a video segment or image by using a textual query.

1 is a flow chart showing a method for learning an information retrieval model based on artificial intelligence according to the present invention.
2 is a flowchart showing an information retrieval method using an artificial intelligence-based information retrieval model according to the present invention.
3 is a diagram schematically showing an information retrieval model according to an example of the present invention.
4 is a diagram schematically showing an apparatus for performing an information retrieval method using an information retrieval model based on artificial intelligence according to the present invention.
5 is a diagram for comparing a search result with a search result of another model according to an embodiment of the present invention.
6 is a view comparing a search result according to an embodiment of the present invention with a search result of another model.
7 is a diagram schematically illustrating a method of generating a passage for searching for a video segment by a search method according to the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same or similar reference numerals are assigned to the same or similar components, and redundant descriptions thereof are omitted. In describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. The accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It is to be understood as including or substituting.

Terms including ordinal numbers, such as first and second, can be used to describe various elements, but these terms are used only for the purpose of distinguishing one element from other elements, and the corresponding elements are these terms. Is not limited by Singular expressions include plural expressions unless the context clearly indicates otherwise.

As used herein, terms such as "comprises", "includes" or "have" should be understood as limiting the existence of features, steps, components, or combinations thereof described in the specification. It is not intended to exclude the possibility of the presence or addition of features, steps, components, or combinations thereof.

Fig. 1 shows a method of learning an artificial intelligence-based IR model according to the present invention, and Fig. 2 shows a step of performing inference using the artificial intelligence-based IR model according to the present invention, 3 schematically shows an example of an information retrieval model according to the present invention. Hereinafter, the present invention will be described with reference to these drawings.

[Learning stage of information retrieval model based on artificial intelligence]

Information retrieval model

In this specification, the information retrieval model is classified into'a retrieval model based on unsupervised methodology' and'a retrieval model based on artificial intelligence'. The former refers to an information retrieval model based on statistical or other unsupervised learning methodologies such as BM25 and QL (Query Likelihood), and the latter is a deep learning series such as DRMM, KNRM, and PACRR, and a language model series such as BERT. It refers to an information retrieval model prepared by learning including, etc.

In the present invention, the retrieval model based on the former unsupervised learning methodology is used to prepare a pseudo-label in the learning stage, and in the inference stage to retrieve a passage from the corpus, and the latter It is used for re-ranking the retrieved passages in the inference stage.

Information retrieval models with open sources are disclosed in the Ad-Hoc Information Retrieval section of Paper With Code.

Search model based on unsupervised learning methodology

As a search model based on an unsupervised methodology, BM25 and QL (Query Likelihood) are known, and Anserini is widely used as an open source. Anserini is a search model that uses RM3, a query expansion methodology, along with BM25. According to research results, the performance of Anserini is similar to that of deep learning search models such as DRMM, KNRM, and PACRR, but it is known to be inferior to that of language model series such as BERT.

The search model based on the unsupervised learning methodology does not require learning and determines the similarity between the query and the document based on statistical theories.

Artificial intelligence-based search model

As a search model based on artificial intelligence, a search model based on deep learning such as DRMM, KNRM, and PACRR, and a search model based on a transformer or language model such as BERT are known. According to the Paper With Code, as of the filing date of this patent, the SOTA (State-of-the-Art) search model has a structure that combines the BERT of the language model and the search model of the deep learning system. It is understood to be.

The artificial intelligence-based search model is completed by learning the similarity between a query and a document based on learning data. In academic terms, AI-based search models are trained using datasets that provide query-document relationships such as TREC, SQuAD, and MS Marco.

Language model search engines such as BERT use Self-supervision learning methodology in the pre-training stage, but deep learning in the fine-tuning stage for information retrieval. Like the search model of, supervised learning based on the data set for the query-document relationship is required.

Weaksupervision methodology

Query-document relational datasets such as TREC, SQuAD, and MS Marco, which are used academically, correspond to ground truths extracted based on actual user search activities, or some use the title of a document as a query. However, there is only a document corpus in the actual field, and there is no query input by the user to search for it, or even if it exists, it is very insufficient to learn an artificial intelligence-based search model. In the present invention, when only a document bundle exists, a pseudo-query is generated from each document, a pseudo-query-document relationship is formed using the generated pseudo-query, and a pseudo-label is generated. In addition, a methodology for learning an artificial intelligence-based search model using this is provided. The WickSupervision methodology for an artificial intelligence-based search model according to the present invention includes the following steps: 1) generating a pseudo-query from each document in a document bundle, 2) the generated pseudo-query Creating a pseudo-query-document relationship by using and generating a pseudo-label based on the pseudo-query-document relationship, 3) Learning an artificial intelligence-based search model using the generated pseudo-label . Hereinafter, each of the above-described steps will be described separately.

1) creating a pseudo-query

One or more keywords are extracted from each document in the document bundle, and this is determined as a pseudo-query. A method of extracting keywords from documents uses a previously known keyword extraction technique. Keyword extraction algorithms can be largely divided into techniques based on unsupervised learning and techniques based on supervised learning, and there are open sources for a number of techniques.

Most of the techniques are methods of extracting a plurality of keywords or phrases from a document, but recently a technique called Doc2Query, which generates natural language sentence-type queries using BERT as a supervised learning-based technique, has also been released as an open source. have.

2) creating a pseudo-label

The pseudo-query extracted from each document is used as an input of a search model based on unsupervised learning methodology such as BM25, and M documents are retrieved from the document stack. In this case, the document from which the pseudo-query is extracted is likely to be included in the upper rank among the M documents, but may not be included.

Among the retrieved M documents, top m (<M) documents are labeled as positive training data, and at least some of the remaining M documents are labeled as negative training data. Depending on the AI-based search model, additional parts can be labeled as neutral training data. In general, positive data and negative data are required, but neutral data is not.

Here, the number of documents M to be retrieved, the number of positive training data m, the number of negative training data, and the number of neutral training data are integers, and may be differently determined according to the characteristics of the document bundle as a kind of hyper parameter. Developers can increase the accuracy of artificial intelligence-based search models by adjusting these hyperparameters in an experimental or theoretical approach according to the characteristics of the document bundle.

3) Learning an artificial intelligence-based search model

Since a query-document relationship is required to train an artificial intelligence-based search model, the training data is provided in the form of a relationship between a query used for retrieving and a document retrieved by the query. For example, it may be given in the form of a pair of two data such as (pseudo-query, positive document), or three data may be provided in a related form such as (pseudo-query, positive document, negative document). Of course, up to a neutral document may be associated with four data.

The amount of data required for learning is not known exactly, and is mainly confirmed by experimental methods. However, the amount of data required for training increases according to the number of parameters included in the model. BERT-series language model-based search models generally have pre-learned models based on Wikipedia, and fine-tuning pre-trained models for specific tasks such as information retrieval is compared with prior learning. Therefore, a small amount of data is required.

In academic research, the amount of learning data is determined according to the available datasets, but in the actual field, the learning data can be determined according to the amount of documents included in the document stack. If the amount of documents included in the document stack is insufficient for learning, another document of a similar field can be added to the document stack, or the amount of data can be increased by using a data augmentation technique.

When training an artificial intelligence-based model, it is generally desirable to divide the data into training data and verification data to verify performance. For example, 80% of the data can be used for training and the remaining 20% of the data can be used for verification.

[Inference stage]

In the present invention, the reasoning step refers to a process of providing a search result in response to a query input by a user. The reasoning step can be broadly divided into the following steps: a retrieving step, a re-ranking step, and an output step. Hereinafter, each step will be described in detail.

Retrieving stage

This is a step of extracting N documents from a bunch of documents to be searched by a search model based on an unsupervised learning methodology. In general, the search model based on artificial intelligence takes considerably longer time for inference than the search model based on unsupervised learning methodology. Therefore, when an artificial intelligence-based search model is applied to all documents included in the document stack, it takes an excessively long time and the user's convenience is degraded. Therefore, after retrieving a number of documents by a search model based on an unsupervised learning methodology, which is relatively fast in the first lane, it is common to apply an artificial intelligence-based search model only to the second retrieved documents. Research to skip the retrieving step by improving the inference speed of the artificial intelligence-based search model is continuously being conducted, but it is understood that it has not yet reached a sufficiently fast inference speed in consideration of user convenience.

At this stage, accuracy is also important, but recall is more important. On the other hand, in the second re-ranking step, accuracy is more important than recall. One of the ways to increase the recall rate is to retrieve the number of documents several times larger than the number of documents to be provided in the search result. Another solution is to increase the reproducibility of the search model based on the unsupervised learning methodology. By using a technique called DeepCT Index using BERT, it is possible to increase the reproducibility while using a search model based on the same unsupervised learning methodology. In order to retrieve a large amount of documents, the time required for retrieving increases in proportion to the number of documents. In the case of using the DeepCT Index technique, it is possible to achieve the same level of reproducibility as in the case of retrieving several times more documents while reducing the time required for searching by reducing the number of documents to be retrieved.

As a search model based on the unsupervised learning methodology, the same search model as used in the learning stage may be used, or another search model may be used. It is important to select a search model with a high reproducibility in the field. In academic research, Anserini (BM25+RM3) is widely used as an open source based search model.

Re-ranking stage

The relevance between the retrieved N documents and the query entered by the user is evaluated by a search model based on the unsupervised learning methodology and sorted according to the degree of relevance. In this step, the relationship between the retrieved N documents and the query entered by the user is re-evaluated by the artificial intelligence-based search model, and then re-ordered, that is, re-ranked. This step is performed by the artificial intelligence-based search model learned in the above-described learning step.

Search results output

The N documents re-ranked by the artificial intelligence-based search model are sorted according to the order of the degree of relevance and output as a search result.

[Processing in special cases]

Matters related to the length of the document

In general, a search model based on unsupervised learning methodology does not have any problem in execution even if the document length is long, but the search model based on artificial intelligence often has a limit on the maximum length of a document that can be processed. In particular, AI-based search models introduced after BERT are limited in the number of maximum tokens that can be processed. For example, in the case of BERT, the maximum number of tokens that can be processed is limited to 512.

When there is a limit on the maximum length of documents that can be processed by the artificial intelligence-based search model, each document in the document corpus can be divided into passages less than the limited length and used as a search target. In this specification, a bundle of passages made of short texts is expressed as a'corpus' in order to distinguish it from a document pile made of a long document.

In the learning stage, it is the same as described above, except that the passages included in the corpus are replaced with documents. That is, a pseudo-query is extracted from a passage, a pseudo-label is created using a search model based on an unsupervised learning methodology, and then an artificial intelligence-based search model is trained. The artificial intelligence-based search model is learned based on the relationship between the pseudo-query and the passage, not the relationship between the pseudo-query and the entire document.

In the inference stage, it is necessary to pre-process each document of the document stack into a corpus by dividing it into a passage, and storing the correspondence between the passage and the document in a form that can be referenced. In the retrieving step, N passages are retrieved from the corpus based on a query input by the user, and in the re-ranking step, the retrieved N passages are reranked.

In the search results, documents other than passages must be provided in order of relevance. To this end, documents are sorted so as to correspond to the sorting order of the re-ranked passage by referring to the correspondence relationship between the passage provided in the preprocessing step and the document, and provided as a search result. Since several passages are separated from one document, the reranked result may include a plurality of passages extracted from one document. In this case, the sorting order of documents may correspond to the order of the most relevant passages.

Document expansion

A long document contains multiple paragraphs, but the content of each paragraph does not necessarily match the subject. Therefore, when each paragraph is divided into passages and stored as a corpus, there is a possibility that the degree of relevance may be evaluated low even though the paragraph is separated from the relevant document.

To prevent this problem, the title of the document or equivalent text can be added to each separate passage in the document. In other words, the same phrase is added to each separate passage in a document. In this specification, the addition of a phrase having a relevance to the passage is expressed as the term'document extension'. The phrase corresponding to the title of the document can be generated, for example, using a text summarization technique. The document summarization technique is largely divided into extractive summarization and abstract summarization, and both can be applied. Techniques for generating additional phrases for document expansion are not necessarily limited to the document summary technique, and any technique capable of compressively expressing the subject of the document may be used.

For example, a query entered by a user includes a plurality of keywords, some of which are included in the original passage, but other keywords may be included in a phrase added by document expansion. Since the text added by the document expansion includes the subject of the entire document, there is a possibility that the passage contains the content intended to be searched by the user. If the text was not added by document expansion, a message that would not have been retrieved can be retrieved by document expansion. This can increase the recall of the retrieving step. As described above, it is important to have high reproducibility in the retrieving step.

If the length of the text added to the passage by document expansion is too long, the length of the passage that is increased by document expansion may exceed the length that can be processed by the artificial intelligence-based search model. In order to minimize the side effects resulting from this, it is desirable that the text added by document expansion be placed in front of the passage. This is because even if the trailing end of a passage lengthened by document expansion is cut off, if the keyword included in the query entered by the user is not included in the trailing end, the passage will still be extracted in the retrieving step.

Domain adaptiveness

When an artificial intelligence-based search model is trained as a document of a specific domain, it can be used as a search model specialized for that domain. For example, in the case of fine-tuning the BERT pre-learned as a general document bundle such as Wikipedia to a document of a specific domain, in the pre-learning stage, the general relationship between the vocabularies is learned, and in fine-tuning, the vocabulary specialized for the domain is learned. You will learn.

A search model specialized for a specific domain may have relatively poor performance for other domains, but it improves performance for that domain.

Ensemble search model

In the above method, the results re-ranked by the artificial intelligence-based search model were used as the final search results, but the search results were provided using an ensemble of search models based on unsupervised learning methodology and search models based on artificial intelligence. May be. In this case, the final evaluation is expressed as in Equation 1.

[Equation 1]

(Final evaluation) = a*(Evaluation of search model based on unsupervised learning methodology)

+ (1-a)*(Evaluation of artificial intelligence-based search model)

In Equation 1, a is a value between 0 and 1, and may be adjusted to provide the best search result as a hyper parameter.

The ensemble model does not necessarily guarantee that the performance is improved compared to the single model, and it can be considered whether or not it is adopted depending on the domain.

4 shows a computer device for performing a search method according to the present invention.

Since the search method and the learning method according to the present invention have been described in detail with reference to FIGS. 1 to 3, an apparatus 100 for performing such a search method will be briefly described with reference to FIG. 4.

Referring to FIG. 4, the computer device 100 includes a processor 110, a nonvolatile storage unit 120 storing programs and data, a volatile memory 130 storing programs being executed, and information between a user and a user. It consists of an input/output unit 140 for inputting and outputting, and a bus, which is an internal communication path between these devices. As a running program, there may be an operating system and various applications. Although not shown, it includes a power supply.

In the learning step, an artificial intelligence-based search model is trained in the memory 130 by using the learning data stored in the storage unit 120. In the inference step, the unsupervised learning-based search model 210 and the artificial intelligence-based search model 220 stored in the storage unit 120 are executed in the memory 130. The corpus is stored in the storage unit 120 and a search method is performed based on a query input through the input/output unit 140.

[Example]

Hereinafter, an embodiment in which the method for providing a search result according to the present invention is applied to a patent document held by the Korea Institute of Industrial Technology will be described.

Artificial intelligence-based search model

As an artificial intelligence-based search model, the open source CEDR-KNRM model was used. In brief, the CEDR-KNRM model is a structure that processes BERT and KNRM in parallel, and details can be checked from related papers and published source codes. According to Paper With Code, it was confirmed that the model had the best performance at the time of implementing the embodiment, and there is no change until the filing date.

Corpus

The number of patent documents to be searched, that is, document bundles, was approximately 3,000, and since the length of all documents exceeded 512 tokens that BERT could process, a corpus was formed by dividing each paragraph. The corpus contains approximately 120,000 passages.

The relationship between each passage and the extracted patent document is stored so that it can be referred to at a later stage of inference.

Create pseudo-query

At least one keyword or phrase was extracted as a pseudo-query by applying the keyword extraction technique to the passage formed by each paragraph of the patent document. The technique used here is RAKE, and the paper and open source are open to the public.

If the same pseudo-query was redundantly removed, approximately 40,000 pseudo-queries were generated.

Query-document relationship formation

Each pseudo-query was input, and 300 passages were extracted from the corpus using BM25, a search model based on the previously known unsupervised learning methodology. Since there were approximately 40,000 pseudo-queries, the total number of extracted passages was 12,000,000. This is an insufficient amount of data to train the CEDR-KNRM model.

Pseudo-labeling

Among 300 passages extracted by each pseudo-query, the upper m passages were classified as positive learning data, and the lower p passages were classified as negative learning data. Since the CEDR-KNRM model also uses neutral training data, the remaining passages excluding the upper m passages and the lower p passages were classified as neutral training data.

Hyperparameters such as m and p were adjusted by verifying the performance of the search model after learning by the inventor reviewing the search results. Since it is a dataset without ground truth, some of the prepared training data can be used as verification data, but since it is a field that can be verified by hand by the inventor, the performance was verified manually.

Reasoning stage

When a user inputs a query composed of a plurality of keywords, 300 passages are first retrieved from the corpus using BM25. The retrieved 300 passages are then provided as a trained CEDR-KNRM model along with a query entered by the user. The CEDR-KNRM model reranks 300 passages. The order of the patent documents is arranged to correspond to the order of the passages by searching the patent documents corresponding to the 300 re-ranked messages. In the case where there is one belonging to the same patent document among the retrieved packages, the corresponding patent documents are sorted according to the order of the highest-valued passages, and the subsequent ones are ignored when sorting the patent documents. Therefore, search results are less than or equal to 300.

Compare search results

In FIGS. 5 and 6, the results searched by the search method developed according to the present embodiment and the results searched by other search methods are described in comparison.

5 is a search result among patents owned by the Korea Institute of Industrial Technology by entering a query including two keywords "wheelchair" and "driving aid", and the left is the search result by the search method developed according to this embodiment, and the right Is the result retrieved from KIPRIS. As shown, in the search method developed according to the present embodiment, documents including words that exactly match the keywords "wheelchair" and "driving aid" as well as other devices that can be used by people with disabilities While patents are also searched, in KIPRIS, only documents containing words that exactly match the keywords "wheelchair" and "driving aid" are searched.

6 is a search result among patents held by the Korea Institute of Industrial Technology by entering a query including six keywords such as "scanning sonar", "safety monitoring", "marine operation equipment", "marine monitoring", and "aquaculture monitoring" As, the upper side is the search result by the search method developed according to the present embodiment, and the lower side is the search result by Google Patent Search. However, in the case of the Google patent search, since the search was not performed when the number of keywords included in the query exceeded 5, the search was performed excluding the keyword "Farm Monitoring". As shown, in the search method developed according to the present embodiment, even if it does not exactly match the keyword included in the query, it searches for a document containing words that are semantically similar, whereas in the Google patent, there is no search result. These results are the same for KIPRIS (not shown).

As can be seen from FIGS. 5 and 6, the search method according to the present invention using an artificial intelligence-based search model does not include keywords included in the input query, but can additionally search documents that are semantically relevant. have.

[Application to video segment search]

7 shows a method of applying the search method according to the present invention to a video segment search.

Text information related to a video includes, for example, an object name recognized by object recognition, a subtitle, and an outline of a video. Among them, the outline of the video is applied to the entire video, and the recognized object name or subtitle is applied only to a specific segment.

In general, the subtitle information is a separate text file, and the subtitle corresponding to the timestamp (or frame number) section is recorded. After reading the subtitle information along with the video, the video player refers to the timestamp and displays the corresponding subtitle on the video.

Various techniques, including YOLO, are known for moving object recognition, and objects are recognized and labeled in near real time. The labeled object name can be saved for the section in which the object appears in a format similar to a subtitle file. If Named Entity Recognition is possible in the object recognition technique, more detailed information may be provided.

As shown in FIG. 7, when the subtitle file and the object information file are integrated and overlapped, the subtitle file and the object information file may be divided so as to be distinguished. Object and subtitle information appearing in each section can be identified. Using the document expansion technique described above, an outline of the video is added to the subtitle information and object information of each section. Then, the information of each section becomes a passage made of text.

These passages form a corpus and can be searched by an input query. For example, the search method according to the present exemplary embodiment may be applied to a case where a scene in which specific objects appear, or a scene in which a specific dialogue appears among movies having a specific content is searched.

[Application to image search]

As in video, object recognition is possible for images. In addition, research on artificial intelligence that generates captions, which are descriptions corresponding to images, is also being conducted steadily. The caption is mainly provided in one sentence, but it is also provided in several sentences.

A corpus passage can be formed by adding the generated caption and recognized object name. However, since the recognized object name is used to generate the caption, adding the caption and the recognized object name may be redundant information. In addition, if there is outline information for a group of images, this outline information may be commonly included in a passage for the group of images.

The passage formed as described above can be searched by the search method according to the present invention.

The above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (10)

As a computer-implemented method that learns an artificial intelligence-based search model for searching video segments by the weak-supervision methodology,
(a) generating a passage corresponding to the video segment including at least one of object information and subtitle information extracted from each video segment;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) generating a pseudo-query from each passage of the corpus formed in step (b) using one or more keywords extracted by a technique based on unsupervised learning or a technique based on supervised learning;
(d) generating a pseudo-label using the corpus generated in step (b), the pseudo-query generated in step (c), and a search model based on an unsupervised methodology; And
(e) learning an artificial intelligence-based search model by using the pseudo-label generated in step (d)
Computer-implemented method for learning an artificial intelligence-based search model for searching for a video segment comprising a.
The method according to claim 1,
Each passage of the corpus further includes an overview of the entire video to which each video segment belongs
A computer-implemented method for learning an artificial intelligence-based search model for searching video segments, characterized in that.
As a device that learns an artificial intelligence-based search model for searching video segments by the weak-supervision methodology,
At least one processor; And
Including at least one memory for storing computer-executable instructions,
The computer-executable instruction stored in the at least one memory, by the at least one processor,
(a) generating a passage corresponding to the video segment including at least one of object information and subtitle information extracted from each video segment;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) generating a pseudo-query from each passage of the corpus formed in step (b) using one or more keywords extracted by a technique based on unsupervised learning or a technique based on supervised learning;
(d) generating a pseudo-label using the corpus generated in step (b), the pseudo-query generated in step (c), and a search model based on an unsupervised methodology; And
(e) learning an artificial intelligence-based search model by using the pseudo-label generated in step (d)
Including
A device that trains an artificial intelligence-based search model to search for video segments.
The method of claim 3,
Each passage of the corpus further includes an overview of the entire video to which each video segment belongs
A device that trains an artificial intelligence-based search model to search for video segments.
As a computer program to provide a method of learning an artificial intelligence-based search model for searching video segments by a weak-supervision methodology,
It is stored in a non-transitory storage medium, and by the processor,
(a) generating a passage corresponding to the video segment including at least one of object information and subtitle information extracted from each video segment;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) generating a pseudo-query from each passage of the corpus formed in step (b) using one or more keywords extracted by a technique based on unsupervised learning or a technique based on supervised learning;
(d) generating a pseudo-label using the corpus generated in step (b), the pseudo-query generated in step (c), and a search model based on an unsupervised methodology; And
(e) learning an artificial intelligence-based search model by using the pseudo-label generated in step (d)
Including
A computer program stored in a non-transitory storage medium that provides a method for learning an artificial intelligence-based search model for searching video segments.
A computer-implemented method for searching for video segments using an artificial intelligence-based search model learned by the method described in claim 1,
(a) generating a passage corresponding to the video segment, including object information and subtitle information extracted from each video segment using an object recognition artificial intelligence model;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) retrieving N passages corresponding to a query input from the corpus formed in step (b) by a search model based on an unsupervised methodology;
(d) re-ranking the N passages retrieved in step (c) based on the input query by the artificial intelligence-based search model; And,
(e) outputting video segments corresponding to the N passages re-ranked in step (d) as search results
Computer-implemented method for searching for a video segment comprising a.
The method of claim 6,
Each passage of the corpus further includes an overview of the entire video to which each video segment belongs
Computer-implemented method for searching for a video segment, characterized in that.
A device for searching for a video segment using an artificial intelligence-based search model learned by the method according to claim 1,
At least one processor; And
Including at least one memory for storing computer-executable instructions,
The computer-executable instruction stored in the at least one memory, by the at least one processor,
(a) generating a passage corresponding to the video segment, including object information and subtitle information extracted from each video segment using an object recognition artificial intelligence model;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) retrieving N passages corresponding to a query input from the corpus formed in step (b) by a search model based on an unsupervised methodology;
(d) re-ranking the N passages retrieved in step (c) based on the input query by the artificial intelligence-based search model; And,
(e) outputting video segments corresponding to the N passages re-ranked in step (d) as search results
An apparatus for searching a video segment comprising a.
The method of claim 8,
Each passage of the corpus further includes an overview of the entire video to which each video segment belongs
Device for searching for a video segment, characterized in that.
A computer program for providing a method of searching for a video segment using an artificial intelligence-based search model learned by the method according to claim 1,
It is stored in a non-transitory storage medium, and by the processor,
(a) generating a passage corresponding to the video segment, including object information and subtitle information extracted from each video segment using an object recognition artificial intelligence model;
(b) forming a corpus from a plurality of video segments into a plurality of passages formed by the step (a);
(c) retrieving N passages corresponding to a query input from the corpus formed in step (b) by a search model based on an unsupervised methodology;
(d) re-ranking the N passages retrieved in step (c) based on the input query by the artificial intelligence-based search model; And,
(e) outputting video segments corresponding to the N passages re-ranked in step (d) as search results
A computer program stored in a non-transitory storage medium for providing a method of searching for a video segment comprising a.

Images