KR20200044644A - Apparatus and method for searching image based on convolutional neural network - Google Patents

Abstract

The present invention relates to a technology which searches an image based on a convolutional neural network (CNN). A characteristic extracted from a fully connected layer of the CNN is used to show an image characteristic, at least one representative image is registered in advance for a text to be searched, and the similarity between the at least one representative image and at least one candidate image is calculated to reflect the calculated similarity in a search ranking. Moreover, a large amount of images can be clustered by using CNN characteristics without predetermining a classification name for search intent.

Description

CNN-based image search method and device {APPARATUS AND METHOD FOR SEARCHING IMAGE BASED ON CONVOLUTIONAL NEURAL NETWORK}

Embodiments relate to a method and apparatus for retrieving an image, and more particularly, to a technique for retrieving an image based on a convolutional neural network (CNN).

The demand to produce and consume images is steadily increasing.

When searching for an image by text, for example, when an image is obtained as a search result by entering specific text in a search box of a search engine, an image included in a document related to the text is exposed as the search result. Since these search results include images that are not related to unnecessary images or text, analysis of the image itself was required to obtain meaningful search results.

As a technology that uses CNN, there is a technology that uses supervised learning-based classification technology for image tagging. For example, Korean Registered Patent No. 10-1908680 discloses a machine learning method and apparatus based on weak supervised learning. However, in the case of such a prior art, sufficient learning data and re-learning are required, and when the model is changed, a large amount of inferencing is required, and thus it is difficult to be utilized for searching for a large amount of images.

As a search result, it is possible to obtain an image search result more in accordance with the user's search intention.

An image retrieval method of a computer device including at least one processor, the method comprising: identifying at least one candidate image included in a search result for a query by the at least one processor; Obtaining, by the at least one processor, a feature value of at least one representative image of a subject corresponding to the query; Image similarity for each of the at least one representative image for each of the at least one candidate image by using the feature values of the at least one representative image and the feature values of the at least one candidate image by the at least one processor Calculating; And reflecting, by the at least one processor, the calculated image similarity in the ranking of the at least one candidate image in the search result.

According to one side, the calculating the image similarity may include calculating the image similarity using a cosine distance between the feature values of the at least one representative image and the feature values of the at least one candidate image. Can be done with

According to another aspect, the step of reflecting in the ranking may include applying the calculated image similarity to text similarity obtained through a text-based search modeling technique for each of the at least one candidate image to score each of the at least one candidate image And re-ranking the basic ranking of the at least one candidate image according to the text similarity according to the calculated score.

According to another aspect, the image search method may use the image-based search model to determine a category corresponding to an image to which the image-based search model is input, using images classified into a plurality of predefined categories according to a user's intention. Learning; And inputting an image selected by the user from among images provided by the user's query into the image-based search model to determine a category corresponding to the selected image as the user's intention.

According to another aspect, the image search method may further include verifying the image-based search model based on the query of the user and click information associated with the selected image.

According to another aspect, the image retrieval method further includes clustering a plurality of images into a plurality of clusters by using the feature values of each of the plurality of images by the at least one processor. The step of checking the candidate image of may be characterized in that images included in the cluster corresponding to the query among the plurality of clusters are identified as the candidate images.

According to another aspect, the step of identifying the at least one candidate image may include storing a query pattern corresponding to each of the plurality of clusters, and in a cluster corresponding to a query pattern extracted for the query among the plurality of clusters. It may be characterized by dynamically weighting.

According to another aspect, the step of identifying the at least one candidate image stores a topic corresponding to each of the plurality of clusters, and dynamically applies to a cluster corresponding to a topic extracted for the query among the plurality of clusters. It may be characterized by giving a weight.

According to another aspect, the step of identifying the at least one candidate image may include dynamically weighting the cluster based on the number of images searched through the query in each of the plurality of clusters. You can.

According to another aspect, in the image search method, when an image is added, the added image is included based on a feature value of the added image and a center value of feature values of images included in each of the plurality of clusters. The method may further include determining a cluster.

According to another aspect, the image search method may further include re-clustering the plurality of images every period or updating the center value.

According to another aspect, each feature value of the at least one representative image and each feature value of the at least one candidate image is CNN (Convolutional) for each of the at least one representative image and for each of the at least one candidate image. Neural Network).

In combination with a computer device, a computer program stored in a computer readable recording medium is provided to execute the image retrieval method on the computer device.

A computer-readable recording medium in which a computer program for executing the image retrieval method is executed is provided.

A computer device comprising: at least one processor implemented to execute instructions readable by the computer device, the at least one processor identifying at least one candidate image included in a search result for a query, Acquiring feature values of at least one representative image for a topic corresponding to the query, and using the feature values of the at least one representative image and the feature values of the at least one candidate image to each of the at least one candidate image It provides a computer device characterized in that for calculating the image similarity for each of the at least one representative image, and reflected in the ranking of the at least one candidate image in the search result using the calculated image similarity.

As a search result, it is possible to provide an image search result more suited to a user's search intention.

1 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
2 is a diagram illustrating an example of a method for calculating image similarity using CNN features in an embodiment of the present invention.
3 is a diagram illustrating an example of a process of re-ranking candidate images in an embodiment of the present invention.
4 is a diagram illustrating an example of a search filter in an embodiment of the present invention.
5 is a diagram illustrating an example of a clustering structure according to an embodiment of the present invention.
6 is a diagram illustrating an example of images clustered in a cluster in an embodiment of the present invention.
7 is a diagram illustrating examples for reflecting a cluster in a search ranking in an embodiment of the present invention.
8 is a diagram illustrating an example of an existing method for excluding duplicates in image search.
FIG. 9 illustrates an improved method for a process for diversification of search results and reduction of a search collection (index size) in one embodiment of the present invention.
10 is a flowchart illustrating an example of an image search method according to an embodiment of the present invention.
11 is a flowchart illustrating an example of a candidate image selection method according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals in each drawing may denote the same members.

The image retrieval method according to embodiments of the present invention may be performed by a computer device to be described later. For example, a computer program may be installed and driven in accordance with an embodiment of the present invention in a computer device, and the computer device may perform an image search method according to an embodiment of the present invention under the control of a driven computer program. You can. The above-described computer program may be stored in a computer-readable recording medium in combination with a computer device to execute an image retrieval method on the computer device. The computer program described herein may have the form of an independent program package, or the form of an independent program package may be installed in a computer device and may have a form associated with an operating system or other program packages.

1 is a block diagram showing an example of a computer device according to an embodiment of the present invention. In one example, the image search method according to the embodiments of the present invention may be executed by the computer device 100 shown through FIG. 1.

1, the computer device 100 may include a memory 110, a processor 120, a communication interface 130, and an input / output interface 140. The memory 110 is a computer-readable recording medium, and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer device 100 as a separate permanent storage device separate from the memory 110. In addition, an operating system and at least one program code may be stored in the memory 110. These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, and memory card. In other embodiments, software components may be loaded into memory 110 via communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer device 100 based on a computer program installed by files received over network 160.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 120 by memory 110 or communication interface 130. For example, the processor 120 may be configured to execute a received command according to program code stored in a recording device such as the memory 110.

The communication interface 130 may provide a function for the computer device 100 to communicate with other devices (eg, the storage devices described above) through the network 160. For example, a request, command, data, file, etc. generated by the processor 120 of the computer device 100 according to program code stored in a recording device such as the memory 110 is controlled by the communication interface 130 through the network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received through the network 160 to the computer device 100 through the communication interface 130 of the computer device 100. Signals, commands, data, etc. received through the communication interface 130 may be transferred to the processor 120 or the memory 110, and files and the like may be further stored by the computer device 100 (described above) Permanent storage device).

The input / output interface 140 may be a means for interfacing with the input / output device 150. For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface 140 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 150 may be configured as a computer device 100 and a single device.

Further, in other embodiments, the computer device 100 may include fewer or more components than those in FIG. 1. However, there is no need to clearly show most prior art components. For example, the computer device 100 may be implemented to include at least some of the input / output devices 150 described above, or may further include other components such as a transceiver, a database, and the like.

The communication method is not limited, and a communication method that utilizes a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network) that the network 160 may include, as well as Bluetooth or NFC (Near Field Communication). Short-range wireless communication such as may also be included. For example, the network 160 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Any one or more of the networks such as the Internet. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. It is not limited.

The feature extracted from the fully connected layer of the CNN can embed the image well, so the CNN feature can be used to represent the image feature. The CNN feature, which embeds an image well, can facilitate image similarity calculations and coupling to probability models.

The fully connected layer is connected to all activations of the previous layer, as in a typical neural network structure. Feature values extracted from the previous convolution layer may be expressed as values of a high-dimensional vector (generally 256 to 4096 dimensions).

At least one representative image may be registered in advance for the text to be searched. At this time, similarity may be calculated between at least one representative image and at least one candidate image, and the calculated similarity may be reflected in the search ranking.

2 is a diagram illustrating an example of a method for calculating image similarity using CNN features in an embodiment of the present invention. For the calculation of image similarity, it is assumed that there is a set R of representative images and a set C of candidate images for a subject (eg, 'apple'). Here, the subject may correspond to a query (or keyword or search term) for searching. FIG. 2 shows an example in which the set R includes text representing an object of the set (eg, 'apple') and n representative images, and the set C includes m candidate images. Here, n and m may be natural numbers. At this time, the image similarity ImageSimilarity ( R , j ) for the set of representative images R of the candidate image j ( j is a natural number less than or equal to m ) may be calculated as in Equation 1 below.

Here, r _i may mean CNN characteristics of the representative image i ( i is a natural number of n or less), and c _j may mean CNN characteristics of the candidate image j , respectively.

In one example, the computer device 100 calculates the image similarity to the set R of the representative image of the candidate image j , CNN features ( r ₁ , r ₂ ,... Of each of the images included in the set R of the representative image. , r _n ) and the CNN feature ( c _j ) of the candidate image j can be extracted to obtain parameters for calculating the image similarity, and the obtained parameters can be reflected in Equation 1 to calculate the image similarity ( R , c _j ). have.

A plurality of representative images may be provided, and the importance may be set differently according to need. For example, the representative image 1 arranged first in the set R may have higher representativeness than other representative images, and in this case, a relatively higher weight may be assigned. In this case, the image similarity for the set R of candidate images j may be calculated as in Equation 2 below.

Here, w _i can be a weight (weight of the CNN of the representative image characteristic r _i i) representative of the image i.

At this time, the computer device 100 may re-rank m candidate images using n representative images.

3 is a diagram illustrating an example of a process of re-ranking candidate images in an embodiment of the present invention. As described above, the representative image 310 may include a topic R corresponding to a query (or keyword or search term) for searching and a set R of at least one representative image. In one example, the administrator may register the subject and set R as the representative image 310 using the management tool 320. In this case, since the administrator only needs to register a small amount of representative images for one subject, the burden of adding a label for individual images can be eliminated.

At this time, the ranker 330 may be implemented to re-rank the images. In one example, when the search server 340 delivers images as a search result for the query to the ranker 330, the ranker 330 processes the re-ranking of the delivered images, and then indexes the re-ranked images ( 350). In this case, the search server 340 may provide search results including images sorted according to the rank of the index 350. Depending on the embodiment, the ranker 330 may be implemented in a form included in the search server 340, or implemented as a separate device from the search server 340 to communicate with the search server 340 through a network. It may be implemented.

At this time, the ranker 330 may calculate the score score ( q , j ) of the image j for the query q using Equation 3 below, and re-rank the images through the calculated score.

Here, the equation (3) for the query image i q Applying ImageSimilarity (R, j) in the image similarity degree for a set of representative images of the image i to the BaseRanking (q, i) text similarity R to the primary rank (summed) to also calculate a score of the image i to query q Is shown. At this time, BaseRanking ( q , i ) indicates the text similarity between the query q and the document including image i , and text-based search modeling techniques (eg, BIM, TF-IDF, which are already known to calculate the similarity) BM25, etc.) may be used. ImageSimilarity ( R , j ) may be calculated as Equation 1 or Equation 2 as described above.

Hereinafter, a method of classifying user requests using a CNN will be described.

4 is a diagram illustrating an example of a search filter in an embodiment of the present invention. The search filter 410 may include search filters according to several predefined categories (clusters) according to a user's intention. For example, the search filter 410 corresponds to a query such as "real life", "color study", "color comic", "sketch", "clipboard", "product", "person", "text", etc. If you select which type of image you want from the images, it can be used to filter and provide only those types of images. In other words, the image search filter may include a function for classifying and filtering images into several predefined categories according to a user's intention. As such, the search filter 410 may classify a type of a user's search request in order to find an exact search result for a user's query, and may be a function for providing images classified by the type.

Meanwhile, a user's intention may be classified using a CNN. For example, as shown in Table 1 below, 8 major search filters can be selected to train a CNN model with 8 labels.

number Classification Learning data Learning data
(augmented) accuracy
(Accuracy) 0 Real picture 54,408 102,312 0.99 One Coloring book 1,059 3,600 One 2 Color cartoon 4,909 18,896 0.862069 3 sketch 1,626 6,108 0.862069 4 Clip art 1,424 5,236 0.785714 5 product 2,618 10,192 0.966667 6 Person 2,782 10,604 0.933333 7 text 1,956 7,520 0.896552

Deep CNN can be used as the CNN model to be learned, and the CNN model can be initialized with ImageNet data. After building the training data, the final training data can be selected through fine-tuning and augmentation. The user intention classification model can be verified based on query and click information associated with the image. For example, the user intention classification model can be verified by checking the actual query of the commonly clicked image based on the click information associated with the image provided as a search result of the image search for the query. For example, click log { click ₁ , click ₂ , click ₃ ,… , click _N }, click _a = ( q , d ) ( N is an arbitrary natural number, a is a natural number less than or equal to N ), and may include image data d accessed through a query q and click. For the image classification {label ₁ , label ₂ ,?, label _l } corresponding to l search filters, the search collection { d ₁ , d ₂ , d ₃ ,… , d _M }, d _b ( M is an arbitrary natural number, b is a natural number less than or equal to M ) may include information about which image classification the corresponding image data is classified into. For example, the image classification L = {0, 1, 2, 3, 4, 5, 6, 7} may correspond to the 8 main search filters described through Table 1 above, and d ₁ is a coloring study and If the image has features classified as a color cartoon, it can be expressed as d ₁ = (1, 2). By joining these click logs and the search collection, {( q ₁ , label _i ), ( q ₂ , label _j ), ( q ₁ , label _k ),… , ( q _x , label _p )} (i, j, k, p is an arbitrary natural number equal to or less than l ), and the relationship between the query and the image classification corresponding to the search filter can be obtained. It can be classified as a set of. At this time, the number of elements in each of the sets may be different from each other. The following is an example of a set for each query.

{( q ₁ , label ₁ ), ( q ₁ , label ₃ ),… , ( q ₁ , label ₇ )}

{( q ₂ , label ₁ ), ( q ₂ , label ₂ ), ( q ₂ , label ₃ )}

…

{( q _k , l ), ( q _k , l ),… , ( q _k , l )}

Through this analysis, the relationship between the query and the search filter can be obtained as shown in Table 2 below as an example.

Search filter ( l ) Incoming query ( q ) Real picture q _One, q _2, q _3, q _4,  q _5, q ₆ Coloring book q _7, q _8, q _9, q _10,  q ₁₁ Color cartoon q _12, q _13, q _14, q ₁₅ sketch q _16, q _17, q _18, q _19,  q ₂₀ Clip art q _21, q _22, q _23, q _24,  q ₂₅ product q _26, q _27, q _28, q _29,  q ₃₀ Person q _31, q _32, q _33, q _34,  q _35, q _36, q _37,  q ₃₈ text q _39, q _40, q _41, q ₄₂

On the other hand, in order to further refine the user's intention, the classification may be extended to eight labels, for example, 37 labels. As an example, Table 3 below shows an example of classifying the search query into 37 image types (op1 to op37). The image type may be classified into individual types by referring to a search term subject (query) classification based on a search log (click log) of the image search.

Main Category Detailed properties Unrealistic op1-13, op21, op36, op37 Iconic op7 (non-reality), op14-20 (reality) Real picture op14-20, op21-35 text op8-10 (non-reality), op33-35 (reality)

The main classification in Table 3 is an example of dividing the user's intention for the user intention classification model into non-reality, iconic, due diligence, and text types, and a more detailed model within each major classification will be developed, as shown in the 37 detailed properties under the main classification. Table 4 below shows examples of detailed search filters according to image types.

Search filter (op1-37) Incoming query ( q ) Sheet Music q _101, q _215, q _34, q _47,  q _45, q ₆₂ drawing q _76, q _123, q _421, q _130,  q ₁₂₁ table q _125, q _133, q _24, q ₁₇ graph q _16, q _172, q _138, q _19,  q ₂₀ BJ Broadcasting q _21, q _212, q _23, q _244,  q ₂₅ map q _256, q _271, q _287, q _249,  q ₃₁₀ … …

In the following, a method of clustering a large number of images using CNN features without specifying a classification name in advance is described.

5 is a diagram illustrating an example of a clustering structure according to an embodiment of the present invention. Clustering can be performed hierarchically. For example, an embodiment in which clustering is performed in 2-depth may be considered. For example, after clustering into 100 clusters, each cluster can be clustered into 100 again. Therefore, it can be expanded to 10,000 clusters. 5, after the images of the image search collection 510 are classified into 100 clusters of the upper level shown in the first dotted box 520, 100 clusters of the lower level included in each of the 100 clusters of the high level. It can be classified as. At this time, the second dotted box 530 represents 100 clusters of the lower level of the 'cluster 2' of the upper level.

Images in the image search collection 510 may be classified using CNN characteristics of each of the images. At this time, the identifier of each cluster may be assigned to indicate the hierarchical structure. For example, the identifier of the lower-level cluster may be assigned by combining the unique number of the higher-level cluster with the unique number in the lower layer. 6 is a diagram illustrating an example of images clustered in a cluster in an embodiment of the present invention.

In the case of such a clustering method, it is possible to classify images very effectively compared to a learning-based technique because it does not need to build a separate training data and can variably determine the number of clusters according to image characteristics in a search collection. In addition, in performing re-ranking (reflecting weights for the clusters containing the most images among the images resulting from the search results for the text (the most clusters)), as the sub-clusters are too subdivided, If a large number of clusters cannot be found (in other words, it is impossible to cover the superiority between clusters in the lower layer classification), the higher layer cluster (cluster of 1depth) in which the corresponding clusters are clustered can be determined as the cluster to reflect the weight. have.

The cluster can be used as a search feature. In other words, the cluster can be reflected in the search ranking. Below, we introduce some methods that can reflect the classified cluster in search ranking.

7 is a diagram illustrating examples for reflecting a cluster in a search ranking in an embodiment of the present invention.

The rule-based search method shown in the first dotted box 710 is a search method in which a suitable cluster is pre-determined according to a query pattern and a cluster is selected according to an extracted query pattern for an input query. (For example, 'cluster 10' is pre-determined as important for query patterns ending with '~ score'). As an example, the rule-based search method constructs a suffix-based query database, extracts a query pattern for an input query through the query database, and dynamically weights a predetermined cluster according to the extracted query pattern It may be a search method that reflects.

The search method of topic modeling shown in the second dotted box 720 determines a topic for each cluster, extracts a topic for a query (e.g., topic 'fashion'), and is dynamic for a cluster for that topic. It may be a search method that reflects the weight (eg, the weight is reflected in clusters 10, 20, and 30 related to the topic 'fashion').

The re-ranking search method shown in the third dotted box 730 dynamically reflects the weight of the cluster (most clusters) containing the most images among the images resulting from the search results for the text. It may be a search method. As an example, the re-ranking search method may be a search method that dynamically reflects weights using the attribute distribution vector of the initial search.

The above-described rule-based search method, the subject modeling search method, and the re-ranking search method may be used separately or may be selectively used according to priority. For example, if a pattern exists in a query, a rule-based search method is used, but if the pattern does not exist, a topic modeling search method is used to extract a topic, and if a topic does not exist, a re-ranking search method is used. Can be utilized. Alternatively, the sum of weights obtained according to the rule-based search method, the subject modeling search method, and the re-ranking search method may be reflected in the search ranking.

When an image to be searched is added, the additional image may be clustered based on the center value of each of the clusters existing in the cluster. For example, the center value of the cluster may be the center value of the CNN feature values of the images included in the cluster, and may belong to a cluster having the center value most similar to the CNN feature value of the additional image. The entire images may be re-clustered at a predetermined period, or the center values of each of the clusters may be updated at a predetermined period.

Hereinafter, a method of excluding similar or duplicate images in a case in which multiple similar images are exposed through a search will be described. FIG. 8 is a diagram illustrating an example of an existing method for excluding duplicates in image search, and FIG. 9 is for an embodiment of the present invention to reduce search results (diversification) and reduce search collections (index size). Describes improved methods for the process.

10 is a flowchart illustrating an example of an image search method according to an embodiment of the present invention. The image search method according to the present embodiment may be performed by the computer device 100 described above as an example. For example, the processor 120 of the computer device 100 may be implemented to execute control instructions according to code of an operating system included in the memory 110 or code of at least one program. Here, the processor 120 is the computer device 100 to perform the steps (1010 to 1050) of the method of FIG. 10, the computer device 100 according to the control command provided by the code stored in the computer device 100 Can be controlled.

In step 1010, the computer device 100 may store at least one representative image for each subject. In one example, text (query / keyword / search term) representing at least one subject and at least one representative image may be stored in the representative image database. The representative image for each subject is an example, and as described through FIG. 3, an administrator may register through the management tool 320.

In operation 1020, the computer device 100 may identify at least one candidate image included in the search result for the query. In one embodiment, candidate images according to a query may include images searched through a search engine included in the computer device 100. In another embodiment, the candidate images according to the query may include images searched through a separate search server, and the computer device 100 may receive candidate images by communicating with the search server in step 1020. . The method of selecting a candidate image will be described in more detail through FIG. 11 later.

In operation 1030, the computer device 100 may obtain feature values of at least one representative image of a subject corresponding to the query. In one example, the computer device 100 may extract at least one representative image stored for a topic corresponding to a query from at least one representative image stored for each topic in step 1010. In this case, the feature values of the extracted at least one representative image may include feature values obtained through a convolutional neural network (CNN) for each of the extracted at least one representative image.

In operation 1040, the computer device 100 determines image similarity for each of the at least one representative image for each of the at least one candidate image by using the feature values of the at least one representative image and the feature values of the at least one candidate image. Can be calculated. For example, the computer device 100 may calculate image similarity using a cosine distance between the feature values of at least one representative image and the feature values of at least one candidate image. Examples of such an image similarity calculation method have been already described through Equations 1 and 2. Equation 2 shows a method of calculating image similarity by assigning weights to representative images. Here, the feature values of the at least one candidate image may include feature values obtained through CNN for each of the at least one candidate image.

In operation 1050, the computer device 100 may reflect the ranking of at least one candidate image in the search result using the calculated image similarity. The re-ranking of candidate images has been described through Equation 3 as an example. For example, the computer device 100 calculates the score of each of the at least one candidate image by applying the calculated image similarity to the text similarity obtained through the text-based search modeling technique for each of the at least one candidate image, and calculates The basic ranking of at least one candidate image according to the text similarity may be re-ranked according to the score. In this case, the re-ranking may include, when a separate basic ranking is not determined, the computer device 100 determines the ranking of each of the candidate images by applying the image similarity to the text similarity.

Meanwhile, in one embodiment, the computer device 100 may utilize an image-based search model to classify user intentions. For example, the computer device 100 learns the image-based search model to determine a category corresponding to the image to which the image-based search model is input using images classified into a plurality of predefined categories according to a user's intention. In addition, an image selected by the user among images provided by the user's query may be input to an image-based search model to determine a category corresponding to the selected image as the user's intention. This image-based search model can be verified based on the user's query and click information associated with the selected image. Previously, an embodiment of associating a query with a category through a click log and a search collection has been described in detail. As an example of such an image-based search model, a CNN model may be used.

As described above, in the present embodiment, the ranking of at least one candidate image included in the search result for the query is re-ranked using image similarity according to the feature values (eg, CNN feature values) of the candidate image and the representative image, It is possible to provide an image more suited to the user's intention. Also, an image-based search model (eg, a CNN model) may be utilized to classify user intentions. The user intention determined for the query may be utilized as a search feature for selecting candidate images according to the query.

Meanwhile, another embodiment for selecting a candidate image will be described later with reference to FIG. 11. 11 is a flowchart illustrating an example of a candidate image selection method according to an embodiment of the present invention. The steps 1110 to 1150 of FIG. 11 may be steps included in the image search method described through FIG. 10. For example, step 1110 may be included before step 1010 or before step 1020, and steps 1120 and 1130 may be included in step 1020. In addition, steps 1140 and 1150 are included after step 1110 and may be performed in parallel with steps 1010 to 1040 of FIG. 10. In another embodiment, the steps 1110 to 1150 of FIG. 11 may be a single image search method. For example, candidate images to be determined may be provided by being included in a search result for a query without re-ranking.

In operation 1110, the computer device 100 may cluster a plurality of images into a plurality of clusters using feature values of each of the plurality of images. At this time, the plurality of clusters may be composed of a plurality of hierarchies (for example, the upper level and the lower level in FIG. 5), as described above.

In operation 1120, the computer device 100 may weight the cluster according to at least one of a rule-based search method, a topic modeling search method, and a re-ranking search method. Here, the rule-based search method may be a search method that stores query patterns corresponding to each of a plurality of clusters and dynamically weights a cluster corresponding to a query pattern extracted for a query among a plurality of clusters. In addition, the search method of subject modeling may be a search method that stores a topic corresponding to each of a plurality of clusters and dynamically weights a cluster corresponding to a topic extracted for a query among a plurality of clusters. Also, the re-ranking search method may be a search method that dynamically weights a cluster based on the number of images included through a query in each of the plurality of clusters. As described above, the search method may be dynamically selected depending on the situation, or two or more search methods may be simultaneously used to sum the weights.

In operation 1130, the computer apparatus 100 may determine images included in the selected cluster as candidate images based on the assigned weight. The determined candidate images may correspond to the candidate images identified in FIG. 10. In another embodiment, when re-ranking of candidate images is not applied, these candidate images may be included in a search result according to a query.

In step 1140, when the image is added, the computer device 100 determines a cluster to include the image to be added based on the feature value of the image to be added and the feature values of the images included in each of the plurality of clusters. You can. In other words, newly added images may also be clustered using feature values.

In operation 1150, the computer device 100 may re-cluster a plurality of images every period or update the center value. The re-clustering or updating of the center value may be performed to reflect that the CNN characteristic value of the cluster is changed by newly added images.

As described above, according to embodiments of the present invention, as a search result, an image search result more suited to a user's search intention may be provided.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor (micro signal processor), microcomputer, field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method for retrieving an image of a computer device including at least one processor, the method comprising:
Checking, by the at least one processor, at least one candidate image included in a search result for a query;
Obtaining, by the at least one processor, a feature value of at least one representative image of a subject corresponding to the query;
Image similarity for each of the at least one representative image for each of the at least one candidate image by using the feature values of the at least one representative image and the feature values of the at least one candidate image by the at least one processor Calculating; And
Reflecting, by the at least one processor, the calculated image similarity in the ranking of the at least one candidate image in the search result.
Image search method comprising a. According to claim 1,
The step of calculating the image similarity,
And calculating the image similarity using a cosine distance between the feature values of the at least one representative image and the feature values of the at least one candidate image. According to claim 1,
Steps reflected in the ranking,
The score of each of the at least one candidate image is calculated by applying the calculated image similarity to text similarity obtained through a text-based search modeling technique for each of the at least one candidate image, and the text similarity is calculated according to the calculated score And re-ranking the basic rank of the at least one candidate image according to the image search method. According to claim 1,
Learning the image-based search model to determine a category corresponding to an image to which an image-based search model is input using images classified into a plurality of predefined categories according to a user's intention; And
Determining a category corresponding to the selected image as the user's intention by inputting an image selected by the user from among images provided by the user's query into the image-based search model
Image search method further comprising a. According to claim 4,
Verifying the image-based search model based on the user's query and click information associated with the selected image
Image search method further comprising a. According to claim 1,
Clustering, by the at least one processor, a plurality of images into a plurality of clusters using feature values of each of the plurality of images.
Further comprising,
Checking the at least one candidate image,
An image search method characterized in that the images included in the cluster corresponding to the query among the plurality of clusters are identified as the candidate images. The method of claim 6,
Checking the at least one candidate image,
An image retrieval method characterized by storing a query pattern corresponding to each of the plurality of clusters, and dynamically weighting a cluster corresponding to a query pattern extracted for the query among the plurality of clusters. The method of claim 6,
Checking the at least one candidate image,
The image retrieval method, characterized in that the subject corresponding to each of the plurality of clusters is stored and the cluster corresponding to the subject extracted for the query among the plurality of clusters is dynamically weighted. The method of claim 6,
Checking the at least one candidate image,
An image search method characterized in that the clusters are dynamically weighted based on the number of images included through the query to each of the plurality of clusters. The method of claim 6,
When an image is added, determining a cluster in which the added image is to be included based on the feature values of the added image and the center values of the feature values of the images included in each of the plurality of clusters.
Image search method further comprising a. The method of claim 10,
Reclustering the plurality of images every period or updating the center value
Image search method further comprising a. According to claim 1,
Each feature value of the at least one representative image and each feature value of the at least one candidate image is obtained through a convolutional neural network (CNN) for each of the at least one representative image and for each of the at least one candidate image. Image search method characterized in that it comprises a feature value. A computer program stored in a computer readable recording medium in combination with a computer device to execute the method of claim 1 on a computer device. A computer-readable recording medium on which a computer program for executing the method of any one of claims 1 to 12 is recorded on a computer device. In the computer device,
At least one processor implemented to execute readable instructions on the computer device
Including,
By the at least one processor,
Identify at least one candidate image included in the search result for the query,
Acquire feature values of at least one representative image for a topic corresponding to the query,
Calculating image similarity for each of the at least one representative image for each of the at least one candidate image by using the feature values of the at least one representative image and the feature values of the at least one candidate image,
Reflecting the ranking of the at least one candidate image in the search result using the calculated image similarity
Computer device characterized in that. The method of claim 15,
By the at least one processor,
Calculating the similarity of the image by using a cosine distance between the feature value of the at least one representative image and the feature value of the at least one candidate image
Computer device characterized in that. The method of claim 15,
By the at least one processor,
The score of each of the at least one candidate image is calculated by applying the calculated image similarity to text similarity obtained through a text-based search modeling technique for each of the at least one candidate image, and the text similarity is calculated according to the calculated score Re-ranking the basic ranking of the at least one candidate image according to
Computer device characterized in that. The method of claim 15,
By the at least one processor,
The image-based search model is trained to determine a category corresponding to an image to which an image-based search model is input by using images classified into a plurality of predefined categories according to a user's intention,
Determining a category corresponding to the selected image as the user's intention by inputting an image selected by the user from among images provided by the user's query into the image-based search model
Computer device characterized in that. The method of claim 15,
By the at least one processor,
Clustering a plurality of images into a plurality of clusters using the feature values of the plurality of images by the at least one processor,
Identifying the images included in the cluster corresponding to the query among the plurality of clusters as the candidate images
Computer device characterized in that. The method of claim 19,
By the at least one processor,
A rule-based search scheme for storing a query pattern corresponding to each of the plurality of clusters and dynamically weighting a cluster corresponding to a query pattern extracted for the query among the plurality of clusters;
A topic modeling search method that stores topics corresponding to each of the plurality of clusters and dynamically weights a cluster corresponding to a topic extracted for the query among the plurality of clusters; And
A re-ranking search method that dynamically weights a cluster based on the number of images included through the query in each of the plurality of clusters
Identifying the candidate image through at least one search method
Computer device characterized in that.

Images