KR102415366B1 - Method and system for retrieving associative image through multimodality ranking model using different modal features - Google Patents

Abstract

A method and system for retrieving related images through a complex ranking model using different modal features are disclosed. The related image search method includes: searching for a document corresponding to an initial query and providing a search result; extracting a plurality of features having different modals when an image is selected from the search result; and retrieving a related image related to the image by using the plurality of features as an input query.

Description

Method and system for searching related images through complex ranking model using features of different modal

The description below relates to techniques for retrieving images.

Most users who use the Internet access the Internet through a browser and then search for information through an Internet portal site. When the user inputs a search term into the search box of the Internet portal, the Internet portal extracts various information corresponding to the search term from the database and provides the search result to the user.

The search results may be divided into categories such as dictionaries, knowledge information, blogs, cafes, specialized materials, sites, books, news, web pages, images or videos, and delivered.

Recently, while various image-based services have appeared, user needs to search for related contents for images found on the Internet are increasing.

For example, in Korea Patent Registration No. 10-1103671 (registration date of January 02, 2012), there is a technology for extracting an image matched with a query from an image group related to the query with respect to a query input by a user and providing it as a search result. has been disclosed.

A general image search is only showing images to which information (eg, tags, names, comments, etc.) that matches a search word input from a user is attached and displayed according to a predetermined standard, and there is a limitation in providing an extended service.

A method and system for providing a related image through an additional image search using a multimodal query as an input query when an image is selected from a search result according to a text-based initial query are provided.

To provide a method and system for quickly searching for a large number of related images with high relevance through a multimodality ranking model using features with different modal in additional image search.

A method for retrieving an associative image executed on a computer system, the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, the method for retrieving an associative image comprising: , providing a search result by searching for a document corresponding to the initial query; extracting, by the at least one processor, a plurality of features having different modals when an image is selected from the search result; and retrieving, by the at least one processor, a related image related to the image by using the plurality of features as an input query.

According to one aspect, the extracting may include extracting a text corresponding to the initial query, a context of a document including the image, and a visual word of the image.

According to another aspect, the extracting may include recognizing, as one of the plurality of features, at least one entity ID from the document through entity recognition technology.

According to another aspect, the extracting may include extracting, as one of the plurality of features, a visual word obtained by quantifying the image.

According to another aspect, the step of extracting the digitized visual word of the image may include digitizing the image through a local descriptor extraction technique or a deep learning-based embedding technique.

According to another aspect, the step of extracting the digitized visual word of the image may include: extracting a CNN feature from the image through a CNN model; and expressing the CNN feature as a word through vector quantization.

According to another aspect, the step of extracting the visual word obtained by digitizing the image may include creating a combination of the visual word through a word concatenate method.

According to another aspect, the plurality of features includes text corresponding to the initial query, an entity ID recognized from a document including the image, and a visual word extracted from the image, and the searching includes: It may include arranging the related image through a ranking model using the text, the entity ID, and the visual word as input queries.

According to another aspect, the searching may include limiting the number of terms corresponding to at least one of the entity ID and the visual word included in the input query.

There is provided a computer program stored in a non-transitory computer-readable recording medium for executing the related image retrieval method in the computer system.

There is provided a non-transitory computer-readable recording medium in which a program for executing the related image search method on a computer is recorded.

A computer system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor comprising: a document search unit that searches for a document corresponding to an initial query and provides a search result; a feature extracting unit for extracting a plurality of features of different modals when an image is selected from the search result; and an image search unit configured to search for a related image related to the image by using the plurality of features as an input query.

According to embodiments of the present invention, when an image is selected from a search result according to a text-based initial query, a related image may be provided through an additional image search using a multimodal query as an input query.

According to embodiments of the present invention, it is possible to quickly search for a large number of highly relevant images through a complex ranking model using features with different modals in additional image search.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating the internal configuration of an electronic device and a server according to an embodiment of the present invention.
3 is a diagram illustrating an example of components that a processor of a server may include according to an embodiment of the present invention.
4 is a flowchart illustrating an example of a method that a server may perform according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a text-based information retrieval environment according to an embodiment of the present invention.
6 is a diagram for explaining a feature used for a related image search according to an embodiment of the present invention.
7 is a diagram for explaining an index structure for a related image search according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a posting list problem in an embodiment of the present invention.
9 is an exemplary diagram for explaining a method of solving a posting list problem in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to techniques for retrieving images.

Embodiments including those specifically disclosed herein can rank images with high relevance through a complex ranking model using different modal features, and through this, more images can be quickly searched for and association between images It is possible to improve the search quality of the related image by improving the image.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , 140 , a plurality of servers 150 , 160 , and a network 170 . 1 is an example for explaining the invention, and the number of electronic devices or the number of servers is not limited as in FIG. 1 .

The plurality of electronic devices 110 , 120 , 130 , and 140 may be a fixed terminal implemented as a computer system or a mobile terminal. Examples of the plurality of electronic devices 110 , 120 , 130 , 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PCs, game consoles, wearable devices, Internet of things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. As an example, in FIG. 1 , the shape of a smartphone is shown as an example of the electronic device 110 , but in embodiments of the present invention, the electronic device 110 is substantially different through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer systems capable of communicating with the electronic devices 120 , 130 , 140 and/or the servers 150 and 160 .

The communication method is not limited, and a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.) that the network 170 may include, as well as a short-distance wireless communication between devices may be included. can For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 170 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110 , 120 , 130 , 140 and the network 170 through a computer device or a plurality of computers that provides commands, codes, files, contents, services, etc. It can be implemented in devices. For example, the server 150 may be a system that provides a first service to a plurality of electronic devices 110 , 120 , 130 , 140 connected through the network 170 , and the server 160 is also a network ( It may be a system that provides the second service to the plurality of electronic devices 110 , 120 , 130 , and 140 connected through 170 . As a more specific example, the server 150 provides a service (eg, a search service, etc.) targeted by the application through an application as a computer program installed and driven in the plurality of electronic devices 110 , 120 , 130 , 140 . The first service may be provided to the plurality of electronic devices 110 , 120 , 130 , and 140 . As another example, the server 160 may provide a service for distributing a file for installing and driving the above-described application to the plurality of electronic devices 110 , 120 , 130 , and 140 as the second service.

2 is a block diagram for explaining the internal configuration of an electronic device and a server according to an embodiment of the present invention. In FIG. 2 , the internal configuration of the electronic device 110 and the server 150 will be described as an example of the electronic device. In addition, other electronic devices 120 , 130 , 140 or server 160 may also have the same or similar internal configuration to the aforementioned electronic device 110 or server 150 .

The electronic device 110 and the server 150 may include memories 211 and 221 , processors 212 and 222 , communication modules 213 and 223 , and input/output interfaces 214 and 224 . The memories 211 and 221 are non-transitory computer-readable recording media, and are non-transitory computer-readable recording media, such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, and the like. It may include a permanent mass storage device. Here, a non-volatile mass storage device such as a ROM, SSD, flash memory, disk drive, etc. may be included in the electronic device 110 or the server 150 as a separate permanent storage device distinct from the memories 211 and 221 . In addition, in the memories 211 and 221 , an operating system and at least one program code (eg, a browser installed and driven in the electronic device 110 or codes for an application installed in the electronic device 110 to provide a specific service) can be stored. These software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memories 211 and 221 through the communication modules 213 and 223 rather than the computer-readable recording medium. For example, the at least one program is a computer program installed by files provided through the network 170 by a file distribution system (eg, the above-described server 160 ) for distributing installation files of developers or applications. It may be loaded into the memories 211 and 221 based on (eg, the above-described application).

The processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223 . For example, the processors 212 and 222 may be configured to execute received instructions according to program codes stored in a recording device such as the memories 211 and 221 .

The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other through the network 170 , and the electronic device 110 and/or the server 150 may communicate with each other through the network 170 . A function for communicating with an electronic device (eg, the electronic device 120 ) or another server (eg, the server 160 ) may be provided. For example, a request generated by the processor 212 of the electronic device 110 according to a program code stored in a recording device such as the memory 211 is transmitted to the server 150 through the network 170 under the control of the communication module 213 . ) can be transferred. Conversely, a control signal, command, content, file, etc. provided under the control of the processor 222 of the server 150 passes through the communication module 223 and the network 170 to the communication module 213 of the electronic device 110 . ) through the electronic device 110 may be received. For example, a control signal, command, content, file, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 or the memory 211 , and the content or file may be transmitted to the electronic device ( 110) may be stored as a storage medium (the above-described permanent storage device) that may further include.

The input/output interface 214 may be a means for interfacing with the input/output device 215 . For example, the input device may include a device such as a keyboard, mouse, microphone, camera, and the like, and the output device may include a device such as a display, a speaker, a haptic feedback device, and the like. As another example, the input/output interface 214 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 215 may be configured as one device with the electronic device 110 . In addition, the input/output interface 224 of the server 150 may be a means for interfacing with a device (not shown) for input or output that is connected to the server 150 or that the server 150 may include. As a more specific example, a service configured using data provided by the server 150 or the electronic device 120 when the processor 212 of the electronic device 110 processes a command of a computer program loaded into the memory 211 . A screen or content may be displayed on the display through the input/output interface 214 .

In addition, in other embodiments, the electronic device 110 and the server 150 may include more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the electronic device 110 is implemented to include at least a portion of the above-described input/output device 215 or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. may include more. As a more specific example, when the electronic device 110 is a smart phone, an acceleration sensor or a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input/output port, and a vibrator for vibration generally included in the smart phone Various components such as may be implemented to be further included in the electronic device 110 .

Hereinafter, a specific embodiment of a method and system for searching a related image through a complex ranking model using different modal features will be described.

3 is a block diagram illustrating an example of components that a processor of a server may include according to an embodiment of the present invention, and FIG. 4 is an example of a method that the server may perform according to an embodiment of the present invention is a flowchart showing

The server 150 according to the present embodiment serves as a platform for providing a search service.

The server 150 provides a document corresponding to the query as a search result for a text-based query input by the user of the electronic device 110 . In this case, the search result may include image content.

When a specific image is selected from the search result according to the text-based initial query, the server 150 may expose the corresponding image through the image viewer, and also an image associated with the image exposed to the image viewer (hereinafter, 'related image') ') can be provided.

The server 150 provides a related image search service that can search for and consume other images highly related to the image from various angles through re-search of the image content.

In particular, the server 150 according to the present embodiment considers the context of text and image corresponding to the initial query for the image currently exposed to the image viewer as it is selected from the search results according to the initial query, and the visual similarity of the image itself. A related image can be serviced through a ranking model.

The processor 222 of the server 150 is a component for performing the related image search method according to FIG. 4 , and as shown in FIG. 3 , a document search unit 310 , a feature extraction unit 320 , and an image search unit It may include a unit 330 . Depending on the embodiment, components of the processor 222 may be selectively included in or excluded from the processor 222 . In addition, according to an embodiment, the components of the processor 222 may be separated or combined to express the functions of the processor 222 .

The processor 222 and the components of the processor 222 may control the server 150 to perform the steps S410 to S430 included in the related image search method of FIG. 4 . For example, the processor 222 and components of the processor 222 may be implemented to execute instructions according to the code of the operating system included in the memory 221 and the code of at least one program.

Here, the components of the processor 222 may be expressions of different functions performed by the processor 222 according to instructions provided by the program code stored in the server 150 . For example, the document search unit 310 may be used as a functional representation of the processor 222 that controls the server 150 according to the above-described command so that the server 150 searches for a document corresponding to the query.

The processor 222 may read a necessary command from the memory 221 in which the command related to the control of the server 150 is loaded. In this case, the read command may include a command for controlling the processor 222 to execute steps S410 to S430 to be described later. Steps S410 to S430 to be described later may be performed in an order different from that shown in FIG. 4 , and some of the steps S410 to S430 may be omitted or additional processes may be further included.

Referring to FIG. 4 , in step S410 , the document search unit 310 responds to the initial query as a search word input as text by the user of the electronic device 110 from the electronic device 110 . The search result for the initial query may be provided to the electronic device 110 by searching for a document. The search result may include at least one document including an image.

In step S420 , the feature extraction unit 320 re-searches a related image related to the target image when an image (hereinafter, referred to as a 'target image') is selected from the search result by the user of the electronic device 110 . features can be extracted for In this case, the feature extractor 320 may generate three features for image re-search from the text corresponding to the initial query used for searching the target image, the context of the document including the target image, and the target image itself.

In step S430 , the image search unit 330 may search for a related image by using three different modal features as an input query for the target image. The image search unit 330 provides a related image search result by arranging images with high relevance to the target image through a complex ranking model using features with different characteristics extracted from text, context, and images as input queries. can

5 is an exemplary diagram for explaining a text-based information retrieval environment according to an embodiment of the present invention.

5 shows an example of the search service screen 500 .

Referring to FIG. 5 , when a search word 501 is input through the search service screen 500 , the processor 222 may use the search word 501 as an initial query and provide a document corresponding to the query as a search result. . In this case, the search result for the initial query is an integrated search result, according to a search method that shows all data such as web documents, news, blogs, images, and videos related to the search word 501 in Internet information search by category.

When a specific image (ie, target image) is selected from the integrated search result according to the initial query, the processor 222 may expose the target image through the image viewer 510, and in addition, An image related to the target image, that is, the related image 520 may be provided.

In other words, the processor 222 provides a related image 520 with respect to the target image currently exposed to the image viewer 510 to provide a service that allows users to search for and consume other images highly related to the image from various angles. do.

The criterion for determining the related image 520 may be defined from three viewpoints as follows.

(1) First, the related image 520 must be related to the first search query used to search for the target image. For example, when a picture of Hong Gil-dong is selected from the search results after searching for 'Hong Gil-dong', it is determined that the image related to the initial query 'Hong Gil-dong' has a high degree of relevance.

(2) Second, it must be related to the event of the target image currently exposed to the image viewer 510 . For example, if the target image is a picture related to a specific drama, it is determined that the picture of 'Hong Gil-dong' classified as an image of the corresponding drama has a high degree of relevance.

(3) Third, it must be an image that is visually similar to the target image currently exposed to the image viewer 510 . Visual similarity is because there is a high probability that the image is included in the same event.

In this embodiment, a ranking model capable of aligning images based on the above three criteria is provided.

Referring to FIG. 6 , the processor 222 may generate a text-based field 601 , a text-based entity name recognition field 602 , and a visual word 603 as features for use in an associated image search. In this case, the text-based field 601 means an initial query used when searching for a target image, and the text-based entity name recognition field 602 means a context that is linguistic information of a document including the target image. , the visual word 603 means a visual vector obtained by digitizing the target image.

7 shows an index structure for a text-based field 601 , a text-based entity name recognition field 602 , and a visual word 603 .

For the text-based field 601, a natural language index structure 701 according to a traditional natural language-based search processing technique is applied.

The text-based entity name recognition field 602 recognizes main entity IDs such as an object (person or thing), a place (location), and an event name (eg, broadcast program name, etc.) in the context of a document containing an image, An index structure 702 according to named entity tagging using text-based word co-existence is applied. As a related entity for each image, a database including all objects, places, event names, etc. is built in advance, and object IDs can be identified by extracting matching objects from these databases.

The visual word 603 applies an index structure 703 according to a technology for digitizing an image. In this case, an inverted index structure may be applied by digitizing the image. A method of extracting a visual word from an image and digitizing the image may use a local descriptor extraction technique or a deep learning-based embedding technique.

In addition to the index structure 703 according to the technology of digitizing an image using deep learning, the visual word 603 uses an index structure 704 using a local descriptor that is a signature of the image according to an embodiment. It is possible. In other words, the visual word 603 may use one of the index structure 703 using a deep learning technique and the index structure 704 using a local descriptor.

In this embodiment, object IDs such as object, place, and event name in the document of the target image using text-based object recognition to consider the context of the image as well as the initial query input by the user for information retrieval is a complex ranking model It can be used as an input query of (610). In addition, in order to consider the similarity of the image itself, it is possible to use the visual word as an input query of the complex ranking model 610 together with text and object ID, including a structure in which an image is digitized as a visual word and indexed.

Accordingly, the composite ranking model 610 may sort the related images using three features having different modalities, that is, text, entity ID, and visual word as input queries.

In deep learning, which is one of the techniques for digitizing images, features can be extracted using a CNN model, and then CNN features can be expressed as words through vector quantization. This method has a problem in that the posting list becomes excessively large due to the limited number of words according to the word vocabulary size.

Referring to FIG. 8 , when the size of the visual word dictionary is 4096 in the index structure 703 using deep learning technology, the document has at least one of 4096 visual words when looking at the posting list. For example, if the number of documents is 6 billion, the average size of the posting list is 6 billion / 4096. As such, when the posting list grows, there is a problem in that it is difficult to generate results within the time given to the search engine at the query time.

In order to solve this problem, it is necessary to increase the size of the word dictionary. In this case, as an example of a method for increasing the size of the word dictionary, a combination of each visual word may be created.

Referring to FIG. 9 , the processor 222 may create a combination of visual words through a concatenation method such as permutation and windowing. When two visual words are combined into one word, the size of the word dictionary can be increased, and through this, the average size of the posting list can be reduced to (number of documents/2 ²⁴ ).

Creating a combination of visual words has the advantage that co-existence of two visual words can be searched with one word. In addition, if priority is applied to the combined visual words according to the strength of each word, similar images can be efficiently found through comparison of fewer words.

Furthermore, in order to guarantee a given response time in a search technique using an inverse index, the number of terms corresponding to features generated in three modals may be partially limited. As an example, the processor 222 may limit the number of terms corresponding to at least one of an object ID and a visual word included in the input query of the complex ranking model 610 . In this case, since the initial query (text) is the most important feature to search for a related image, it can be excluded from the subject of limiting the number of terms.

As such, according to embodiments of the present invention, when an image is selected from a search result according to a text-based initial query, a related image may be provided through an additional image search using a multimodal query as an input query. And, according to embodiments of the present invention, it is possible to quickly search for a large number of highly related images through a complex ranking model using features with different modals in additional image search.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store a program executable by a computer, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed over a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method for retrieving a related image executed in a computer system, the method comprising:
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The related image search method includes:
providing, by the at least one processor, a search result by searching for a document corresponding to an initial query;
extracting, by the at least one processor, a plurality of features of different modals when an image is selected from the search result; and
retrieving, by the at least one processor, a related image related to the image using the plurality of features as an input query;
including,
The step of extracting the plurality of features comprises:
extracting a visual word obtained by digitizing the image; and
creating a combination of the visual word as one of the plurality of features;
A related image search method comprising a. According to claim 1,
The step of extracting the plurality of features comprises:
extracting, as one of the plurality of features, the text corresponding to the initial query, and the context of a document including the image;
A related image search method further comprising a. According to claim 1,
The step of extracting the plurality of features comprises:
recognizing, as one of the plurality of features, at least one entity ID from the document through entity recognition technology;
A related image search method further comprising a. delete According to claim 1,
The step of extracting the visual word obtained by digitizing the image is
Numericalizing the image through local descriptor extraction technology or deep learning-based embedding technology
A related image search method comprising a. According to claim 1,
The step of extracting the visual word obtained by digitizing the image is
extracting CNN features from the image through a CNN model; and
Expressing the CNN feature as a word through vector quantization
A related image search method comprising a. According to claim 1,
The step of creating a combination of the visual word is,
Creating a combination of the visual words through a word concatenate method
A related image search method comprising a. According to claim 1,
The searching step is
Sorting the related images through a ranking model using the plurality of features as input queries
A related image search method comprising a. 9. The method of claim 8,
The searching step is
limiting the number of terms corresponding to at least one of the plurality of features included in the input query;
A related image search method comprising a. A computer program stored in a non-transitory computer readable recording medium for executing the method of any one of claims 1 to 3, 5 to 9, in the computer system. [Claim 10] A non-transitory computer-readable recording medium in which a program for executing the related image retrieval method of any one of claims 1 to 3 and 5 to 9 on a computer is recorded. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
a document search unit that searches for a document corresponding to the initial query and provides a search result;
a feature extracting unit for extracting a plurality of features of different modals when an image is selected from the search result; and
An image search unit that searches for a related image related to the image by using the plurality of features as an input query
including,
The feature extraction unit,
Extracting the visual word digitizing the image,
one of said plurality of features, making a combination of said visual words;
A computer system characterized by a. 13. The method of claim 12,
The feature extraction unit,
Extracting the text corresponding to the initial query and the context of a document including the image as one of the plurality of features
A computer system characterized by a. 13. The method of claim 12,
The feature extraction unit,
one of said plurality of features, recognizing at least one entity ID from said document through entity recognition technology;
A computer system characterized by a. delete 13. The method of claim 12,
The feature extraction unit,
Numericalizing the image through local descriptor extraction technology, or deep learning-based embedding technology
A computer system characterized by a. 13. The method of claim 12,
The feature extraction unit,
After extracting the CNN features from the image through the CNN model,
Expressing the CNN feature as a word through vector quantization
A computer system characterized by a. 13. The method of claim 12,
The feature extraction unit,
Creating a combination of the above visual words through a word merging method
A computer system characterized by a. 13. The method of claim 12,
The image search unit,
Sorting the related image through a ranking model using the plurality of features as an input query
A computer system characterized by a. 20. The method of claim 19,
The image search unit,
limiting the number of terms corresponding to at least one of the plurality of features included in the input query
A computer system characterized by a.

Images