KR102130448B1 - Method, apparatus and computer program for searching image - Google Patents

Abstract

In the present invention, in searching for an image similar to the search image input by the user, the one or more search images based on each search criterion attribute using one or more search images input by the user and respective search criterion attributes therefor An image retrieval method, device, and computer program characterized by retrieving similar images by combining. More specifically, the server receives a plurality of retrieval data information including one or more retrieval images input by a user from a terminal. Receiving search information; A similar image search step of searching for a similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And a similar image providing step of transmitting the similar image information on the searched similar image to the terminal.

Description

Method, apparatus and computer program for searching image

The present invention relates to a method for searching an image, a device, and a computer program, and more specifically, by using a plurality of search data information and a plurality of search attribute information therefor, including one or more search images input by a user, It relates to an image search method, apparatus and computer program characterized by searching for similar images by combining parts corresponding to the search attribute information in each search data information.

2. Description of the Related Art With the development of information communication technology, search for information using various terminals such as a smart phone or personal computer (PC) is widely used.

Particularly, in the past, documents including the text are searched and provided mainly based on text such as a search word input by a user, but in recent years, users are increasingly searching for images such as photos or pictures. This is because it is often more efficient to look at images such as actual photographs than to acquire information only with text, as in the words 百聞不如一見). However, if it is not easy to clearly describe the characteristics with text only, such as clothing or scenery, the user may have difficulty finding the image he or she wants. For example, even when a user wants to search for clothing in a style that he thinks, it is often encountered when entering text as a search term makes it difficult to find an appropriate image.

On the other hand, recently, a service has been attempted to receive a search image, not text, from a user, and to search and provide an image similar to the input search image. However, in this case, only the image most similar to the search image input by the user may be selected and provided, and a similar image may not be searched and provided based on specific attributes of the search image. That is, in the related art, it was possible to search for the most similar image in consideration of the shape, color, texture, etc. of objects included in the input search image, but a similar image is searched based on specific attributes of the input search image. Or, it was difficult to search for an image based on a combination of specific attributes of a plurality of inputted search images.

For example, in the related art, when a user inputs a red horizontal stripes T-shirt image as a search image, it is possible to search for an image most similar to the input red horizontal stripes T-shirt image, but similar to the T-shirt included in the search image. It was difficult to search for images of T-shirts with different colors or different patterns, and further, a second search was made for the T-shirt form of the first search image by combining the T-shirt form of the first search image and the pattern pattern of the second search image. As in the case of searching for an image having a pattern pattern of an image, it is more difficult to search for a similar image by combining properties of objects included in various search images.

Accordingly, in searching for an image, not only can an image be searched and provided based on a specific attribute of a given search image, but also a method of searching and providing a similar image by combining specific attributes of a plurality of search images This would allow the user to search for the desired image more conveniently and accurately, but a suitable solution for this has not yet been provided.

Republic of Korea Patent Publication No. 10-2007-0009338 (published on January 18, 2007)

The present invention was created to solve the problems of the prior art as described above, and in searching and providing an image similar to the search image input by the user, the similar image is based on a specific attribute of the search image input by the user. It is an object of the present invention to provide an image search method, device, and computer program that can be searched and provided.

Furthermore, an object of the present invention is to provide an image search method, apparatus, and computer program capable of searching for and providing a similar image by combining specific attributes of a plurality of search images input by a user.

An image search method according to an aspect of the present invention for solving the above problems includes: a search information receiving step in which a server receives a plurality of search data information including one or more search images input by a user from a terminal; A similar image search step of searching for a similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And providing a similar image for transmitting the similar image information for the searched similar image to the terminal.

In addition, the image search method according to another aspect of the present invention, the search information transmitting step of transmitting a plurality of search data information including at least one search image input by the user to the server; A similar image information receiving step of receiving, from the server, similar image information for the searched similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And providing similar image information to provide the user with similar image information about the similar image.

A computer program according to another aspect of the present invention is characterized by being a computer program stored in a computer-readable medium for executing each step of the image retrieval method described above in a computer.

A server according to another aspect of the present invention, a search information receiving unit for receiving a plurality of search data information including one or more search images input by a user from a terminal; A similar image search unit for searching for a similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And a similar image providing unit that transmits similar image information on the searched similar image to the terminal.

A terminal according to another aspect of the present invention, a search information transmitting unit for transmitting a plurality of search data information including one or more search images input by a user to a server; A similar image information receiving unit configured to combine the search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information from the server and receive similar image information for the searched similar image; And a similar image information providing unit that provides similar image information for the similar image to the user.

In an image search method, apparatus, and computer program according to an embodiment of the present invention, in searching for and providing an image similar to a given search image, a user searches and provides a similar image based on a specific attribute of a given search image, so that the user You can search for the image you want more conveniently and accurately.

In addition, in the image search method, apparatus, and computer program according to an embodiment of the present invention, by searching and providing an image similar to an image combining the plurality of search images based on respective search attributes for the plurality of search images, It is possible to more conveniently and accurately receive a similar image for an image combining the plurality of search images.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention and describe the technical spirit of the present invention together with the detailed description.
1 is a configuration diagram of an image search system according to an embodiment of the present invention.
2 is a flowchart of an image search method from a server perspective according to an embodiment of the present invention.
3 is an exemplary view of a search information input screen according to an embodiment of the present invention.
4 is a diagram illustrating a result of a similar image search using a plurality of search images according to an embodiment of the present invention.
5 is an exemplary view of a search information input screen according to another embodiment of the present invention.
6 is a diagram illustrating a result of a similar image search using a search image and text input according to an embodiment of the present invention.
7 is an exemplary view of a search information selection screen according to an embodiment of the present invention.
8 and 9 are detailed flowcharts of an image search method according to an embodiment of the present invention.
10 to 12 are diagrams for explaining a specific operation of the image search method according to an embodiment of the present invention.
13 is a view for explaining the operation of the image search system according to an embodiment of the present invention.
14 to 15 are diagrams illustrating a method of calculating an index feature vector according to an embodiment of the present invention.
16 is a flowchart of an image search method from a terminal perspective according to an embodiment of the present invention.
17 is a specific flowchart of an image search method according to an embodiment of the present invention.
18 is a configuration diagram of a server for image search according to an embodiment of the present invention.
19 is a configuration diagram of a terminal for image search according to an embodiment of the present invention.

The present invention can be applied to various conversions and can have various embodiments. Hereinafter, specific embodiments will be described in detail with reference to the accompanying drawings.

In the description of the present invention, when it is determined that a detailed description of known technologies related to the present invention may obscure the subject matter of the present invention, the detailed description will be omitted.

Terms such as first and second may be used to describe various components, but the components are not limited by the terms, and the terms are only for the purpose of distinguishing one component from other components. Is used.

Hereinafter, exemplary embodiments of the image search method, apparatus, and computer program according to the present invention will be described in detail with reference to the accompanying drawings.

First, FIG. 1 shows a configuration diagram of an image search system 100 according to an embodiment of the present invention. As can be seen in FIG. 1, the image search system 100 according to an embodiment of the present invention transmits a plurality of search data information including one or more search images input by a user to the server 120, A plurality of search data information including the terminal 110 that receives the similar image information for the similar image searched from the server 120 and provides it to the user, and one or more search images input by the user from the terminal 110 When received, the similar data is searched by combining the search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information, and the similar image information for the similar image is received by the terminal ( It may be configured to include a server 120 for transmitting to 110 and the network 130 connecting the terminal 110 and the server 120.

At this time, as the terminal 110, various portable terminals such as a smart phone, a tablet PC, a PDA, and a mobile phone can be used. In addition, the user receives search information from a user, such as a personal computer (PC) or notebook PC, and receives the search information. Various types of terminals that can be used to provide a searched image can be adopted.

In addition, in the present invention, the image includes a picture, a photo, and the like, and may be configured in various ways, including letters, symbols, and shapes.

In addition, the network 130 connecting the terminal 110 and the server 120 may include a wired network and a wireless network. Specifically, a local area network (LAN), a metropolitan network (MAN) Area Network (WAN), Wide Area Network (WAN), and may include various communication networks. In addition, the network 130 may include a known World Wide Web (WWW). However, the communication network 130 according to the present invention is not limited to the networks listed above, and may include at least a part of a known wireless data network, a known telephone network, or a known wired/wireless television network.

2 is a flowchart of an image search method from the perspective of the server 120 according to an embodiment of the present invention. As can be seen in Figure 2, the image search method according to an embodiment of the present invention, the server 120 receives a plurality of search data information including one or more search images input by the user from the terminal 110 Search information receiving step (S110), a similar image search step of searching for a similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information (S120) And it may include a similar image providing step (S130) of transmitting the similar image information for the searched similar image to the terminal.

Hereinafter, an image retrieval method according to an embodiment of the present invention is divided into steps and reviewed in detail with reference to FIG. 2.

First, in the search information receiving step (S110), the server 120 receives a plurality of search data information including one or more search images input by a user from the terminal 110. At this time, the server 120 may receive a plurality of search attribute information for the plurality of search data information together with the plurality of search data information from the terminal 110.

As a specific example, as can be seen in Figure 3, the user inputs a plurality of search images (101, 102, 103) to the terminal 110, and searches for the plurality of search images (101, 102, 103) Data information may be transmitted to the server 120.

At this time, the plurality of search images (101, 102, 103) itself may be transmitted as the search data information, but the URL information of the plurality of search images (101, 102, 103) is transmitted and the server 120 The plurality of search image (101, 102, 103) files may be collected from a device other than the terminal 110 using the URL information.

In addition, the user may input the search images 101, 102, and 103 in a variety of ways, such as directly photographing them using his terminal or selecting an image created by himself or another user.

In addition, the user may input search text other than the search images 101, 102, and 103 as the search data information.

In addition, search attribute information (for example, "shape", "pattern", "color" in FIG. 3) for a plurality of search data information, such as the search image, is set in advance and a user searches for each search attribute information An image may be input, or a user may set search attributes for the inputted search image after inputting the search image or the like. For example, in FIG. 3, a user sets "form" as the search attribute information for the first search image 101 (104 in FIG. 3), and "pattern as the search attribute information for the second search image 102" &Quot;(105 in FIG. 3)", and as the search attribute information for the third search image 103, "color" is set (106 in FIG. 3).

Subsequently, in the similar image search step (S120 ), the server 120 searches for a similar image by combining search data partial information corresponding to each search criterion attribute information from each search data information.

That is, referring to the case of FIG. 3, since the search attribute information of the first search image 101 input by the user is “shape”, the first search image 101 excludes “pattern” or “color”. And take the information about the "shape" of the shirt included in the first search image 101, similarly take the "pattern" information in the second search image 102, and in the third search image 103" Similar to the image combining "color" information and combining "shape" of the first search image 101 and "pattern" of the second search image 102 and "color" of the third search image 103 The image will be searched.

Accordingly, as can be seen in FIG. 4, when a plurality of search images ((a), (b), (c) in FIG. 4) are input, a plurality of search criteria attribute information for the plurality of search images ( For example, an image combining the plurality of search images based on "shape", "pattern", and "color" ("shape" of the first search image 101 and "of the second search image 102") Patterns and similar images for the "image" having the "color" of the third search image 103 are searched.

Accordingly, as can be seen in (d) of FIG. 4, the “shape” of the first search image 101 and the “pattern” of the second search image 102 and the “color” of the third search image 103 It can be seen that similar images including the characteristics of "can be searched.

Subsequently, in the step of providing the similar image (S130 ), the server 120 transmits the similar image information for the searched similar image to the terminal 110.

At this time, the server 120 may transmit the searched similar image itself as the similar image information, but the URL information of the searched similar image is transmitted, so that the terminal 110 uses the URL information or the like to transmit the similar image. The server 120 may be collected from other devices.

In addition, the present invention will be mainly described with reference to examples such as "clothing" below, but the present invention is not limited to this. In addition, the present invention combines specific attributes of a plurality of search data information such as search images and search keywords. It can be used in a variety of applications to search for similar images.

In addition, the image search method according to an embodiment of the present invention may not only search for a similar image using a plurality of search images input by the user, but also search for one or more search images and one or more search text input by the user. You can also search for similar images.

For example, FIG. 5 illustrates a case in which the user inputs the second search text 301 together with the first search image 302.

At this time, in the step of receiving the search information (S110), the server 120 receives the first search image 302 input by the user from the terminal 110 and the first search attribute information 305 for the user. 2 Search information including the search text 301 and second search criterion information 304 therefor is received. Accordingly, in the similar image search step (S120), the server 120 searches for the first search image. At 302, first search data partial information (ie, “pattern” of the first search image 302 in FIG. 5) corresponding to the first search attribute information (“pattern” in 305 in FIG. 5), and The second search text part 301 is combined with the second search data part information (ie, the "shape" of the skirt) corresponding to the second search attribute information ("shape" in 304 of FIG. 5), and is shown in FIG. Similar images are searched for an image combining the "pattern" of the first search image 302 and the "shape" of the skirt.

In addition, as shown in FIG. 5, since the input of the search image or search text for “color” (306 in FIG. 5) is omitted (303 “pass” in FIG. 5), “color” in searching for a similar image Similar images can be searched on the basis of "shape" and "pattern" without limitation.

Accordingly, as can be seen in FIG. 5, when one first search image ((a) of FIG. 5) and one second search text (“skirt”) are input, the first search image and the second Based on each search criteria attribute information for the search text (in FIG. 4, “pattern” 305 for the first search image 302, “form” 304 for the second search text 301) Search for a similar image for an image that combines partial information of the first search image and the second search text (the image having the "pattern" of the first search image 302 and the "shape" of the second search text 301) Is done.

Furthermore, when a search image is input in the present invention, after detecting a predetermined object among the search images, an image search may be performed using the detected object. As shown in FIG. 6(a), if an area that does not correspond to an object among the inputted search images exists, this may interfere with obtaining a search image desired by the user. After the detection (401 in FIG. 6), if the image search is performed using the detected object area (FIG. 6 (b)), a more accurate search result can be derived.

Accordingly, as can be seen in (c) of FIG. 6, the “shape” of the first search image 101 and the “pattern” of the second search image 102 and the “color” of the third search image 103 It can be seen that similar images including the characteristics of "can be searched.

In addition, in the present invention, when only the search image and search text for "shape" and "pattern" are input and no search image or search keyword for "color" is input, it can be seen in FIG. 6(d). As shown, it can be seen that a skirt having a similar pattern according to the inputted search image and search text may be searched even if it has a completely different color.

In addition, in the image search method according to an embodiment of the present invention, the terminal 110 may provide a user interface through which the user can select a previously stored search image and search attribute information on the search image. The user may search for a similar image more conveniently using the previously stored search image and search attribute information for the search image.

More specifically, as can be seen in FIG. 7, the terminal 110 may provide the “pattern” list 502 that the user has previously stored so that the user can select 201. To this end, when a user takes a picture of an object, an interface for selecting search attribute information for the object (for example, a menu for selecting "shape", "pattern", "color", etc.) is provided. Then, the search attribute information selected by the user and the picture may be stored in the terminal 110 or transmitted to the server 120 for storage, and may be easily selected by the user later. At this time, the picture or the like may be stored as the image itself, but may be converted into a feature representing the characteristic or the like and stored.

Furthermore, while providing a search image and search attribute information to the user, a function of recommending the user using the user's selection history or taste, profile, and statistics may be added.

For example, if a user frequently searches for many pieces of a dot pattern, even if the user does not specify a search image or search text for a pattern item, the search result may be recommended to the user by adding a dot pattern condition. Alternatively, the search result may be recommended to the user by using an item corresponding to a trendy item searched by users a lot in the last month and a search condition therefor. In this case, items searched by users a lot may be selected as targets of behavior patterns of all other users who use the service, and may be selected in consideration of behavior patterns of users similar to the age, gender, and taste of the target users. have.

Furthermore, the user may designate a target product list of an image to be searched so that an image search is performed within the specified category range. For example, the user may specify a list of items to be searched through text input or list selection, such as "clothing" or "furniture", and in this case, in the present invention, the search image is within the range of the specified list of items to be searched for. B. An image is searched by using a plurality of search data information such as search text and a plurality of search attribute information therefor.

Further, in the image search method according to an embodiment of the present invention, after calculating a feature vector for each of a plurality of search image attributes corresponding to the plurality of search attribute information from the inputted plurality of search images, the calculated search image Similar images can be searched from a plurality of stored images stored in the database 124 using the feature vector for each attribute.

At this time, the database 124 stores the feature vectors for each of the plurality of stored image attributes calculated and stored based on the plurality of search attribute information for the plurality of stored images together with the plurality of stored images. Accordingly, in the image retrieval method according to an embodiment of the present invention, a feature vector by a plurality of search image attributes calculated from the plurality of search images is compared with a feature vector by a plurality of stored image attributes stored in the database 124. Accordingly, an image similar to an image combining the plurality of search images may be searched and provided based on each search attribute information for the plurality of search images.

More specifically, as shown in FIG. 8, in the image search method according to an embodiment of the present invention, the similar image search step (S120) corresponds to the plurality of search attribute information in the plurality of search images. A plurality of search image attribute-specific feature vector calculation steps (S121) and a plurality of stored images stored in the database 124 are calculated based on the plurality of search attribute information and stored in a plurality of search image attribute-specific feature vectors to calculate the feature vectors It may include a database search step (S122) of searching for a similar image by comparing the feature vector for each stored image attribute with the feature vector for each of the searched image attributes.

Furthermore, as can be seen in FIG. 9, the feature vector calculating step (S121) for each attribute of the search image includes detecting one or more objects included in the search image (S1211) and the one or more objects, respectively. The step of calculating a feature vector for each search image attribute based on the search attribute information may be included (S1212 ).

Hereinafter, in the image search method according to an embodiment of the present invention with reference to FIGS. 10 to 12, a feature vector for each search image attribute is calculated from a plurality of search images, and among them, among the plurality of stored images stored in the database 124 The method of searching for similar images will be described in more detail.

First, when a user inputs a method of photographing or uploading the search image 701 (FIG. 10(a)), the search image 702 is trained using a machine learning technique such as deep learning. It is input to the neural network model and goes through the processing process.

At this time, the neural network model is constructed based on a Region with CNN features (R-CNN) algorithm, which is a kind of deep learning technology, and accordingly detects the location of an object such as a fashion item in the inputted search image 701 It may be implemented as an object detector (object detector) model (702, 703, 704 in FIG. 10(b)), and further, the object may be classified according to a predetermined type. More specifically, as the object detector model, a YOLO v2 model or the like can be used, wherein the YOLO v2 model can be implemented by learning using learning data such as a fashion item image having a bounding box. have.

Furthermore, as an embodiment of the present invention, the neural network model may classify the detected fashion items, such as objects, into tops, bottoms, bags, shoes, outers, etc. after detecting the position of the objects such as fashion items in the input image. .

In addition, in an image search method according to an embodiment of the present invention, a shape, texture, color, and the like in advance for an object, such as a fashion item included in the search image 701, using a neural network model in advance It can be implemented as a feature vector extractor (feature vector extractor) model for calculating a feature vector for each search image attribute according to the specified search attribute information, and accordingly, as shown in FIG. 10(c), the detected object ( With respect to 705, 706, and 707, a feature vector 708 for each search image attribute according to predetermined search attribute information such as category, texture, and color may be calculated.

More specifically, as the feature vector extractor model, an Inception v3 model or the like can be used, wherein the Inception v3 model uses a pre-trained model to create a category, texture, and color. It can be implemented by performing fine tuning with learning data such as a fashion item image according to predetermined search attribute information.

Furthermore, the learning data may be produced by tagging the location and search attribute of an object in a fashion item image collected through web crawling or the like.

For a more specific example, when the first search image 801 of FIG. 11 is input, the objects 802, 803, and 804 included in the first search image may be detected, and then, for each of the objects in advance A feature vector for each search image attribute according to predetermined search attribute information is calculated. In the first search image of FIG. 11, objects of the top 802, bottom 803, and bag 804 were detected, and then a feature vector for each of the search image attributes for the top 802 according to the "form" search attribute information. (802a) was calculated ({0.22, 0.8280, 0.3311, ...}), and the characteristic vector 803a for each search image attribute for the bottom 803 was calculated ({0.5213, 0.0027, 0.68, ... }) Also, it can be seen that the characteristic vector 804a for each search image attribute for the bag 804 has been calculated ({0.48, 0.0187, 0.241, ...}).

Also, a feature vector for each search image attribute according to search attribute information such as "pattern" and "color" is calculated from other search images input by the user. For example, for the second search image, a feature vector for each search image attribute according to “pattern” search attribute information is calculated (eg, {0.761, 0.8616, 0.135, ...}), and the third search image is For, after calculating a feature vector for each search image attribute according to the "color" search attribute information (for example, {0.101, 0.1672, 0.583, ...)), search for a "shape" for the top of the first search image Feature vector ({0.22, 0.8280, 0.3311, ...}) per search image attribute according to attribute information and feature vector per search image attribute according to "pattern" search attribute information for the second search image ({0.761, 0.8616) , 0.135, ...}) and a feature vector ({0.101, 0.1672, 0.583, ...}) for each search image attribute according to "color" search attribute information for the third search image ([{0.22, 0.8280, 0.3311, ...}, {0.761, 0.8616, 0.135, ...}, {0.101, 0.1672, 0.583, ...}], using this, the "shape" of the first search image and the second search image It is possible to effectively search similar images similar to the image in which the "pattern" of and the "color" of the third search image are combined.

Furthermore, for the stored image stored in the database 124, as can be seen in FIG. 12, after detecting the objects included in the stored image, the detected object (901 in FIG. 12) in advance The feature vector for each stored image attribute according to the determined search attribute information is calculated.

Accordingly, for the object 901 detected in the stored image of FIG. 12, a characteristic vector 902 for each stored image attribute was calculated according to predetermined search attribute information “shape”, “pattern”, and “color” (shape: {0.22, 0.8280, 0.3311, ...}, pattern: {0.342, 0.081, 0.0017, ...}, color: {0.84, 0.5402, 0.7813, ...}).

Accordingly, the characteristic vector for each stored image attribute for each search attribute information is stored in the database 124 (903 in FIG. 12) and can be used to search for a similar image for an image combining a plurality of search images.

Further, the database 124 may store a connection storage image feature vector to which the feature vectors for each of the plurality of stored image attributes are connected to the stored image, in which case the server 120 may feature the plurality of search image attributes. After generating a linked search image feature vector to which a vector is connected, a similar image can be searched based on the similarity with the linked stored image feature vector stored in the database 124.

Accordingly, by combining feature vectors for each of the search image attributes for the plurality of search images, a stored image having a feature vector for each stored image attribute similar to the feature vector for each combined search image attribute is searched for a plurality of search images. With respect to each search attribute, it is possible to effectively search for and provide an image similar to an image combining the plurality of search images.

In addition, FIG. 13 is a diagram for explaining in detail the operation of the image search method, the apparatus and the system 100 according to an embodiment of the present invention.

Hereinafter, the operation of the image search method, apparatus, and system 100 according to an embodiment of the present invention will be described in more detail with reference to FIG. 13.

First, as shown in FIG. 13, a user may input search data information such as a plurality of search images 601, 602, and 603 using the terminal 110.

As a more specific example, a case in which the user inputs the first search image 101, the second search image 102, and the third search image 103 as shown in FIG. At this time, for some or all of the plurality of search images 601, 602, and 603, as described in FIG. 6(b), a process of detecting an object (for example, a fashion item, etc.) may be performed. (604).

Subsequently, a neural network model or the like is applied to the plurality of search images 601, 602, and 603 to detect predetermined search attribute information such as category, texture, and color. A feature vector for each search image attribute is calculated (605, 606, 607).

Accordingly, similar images may be searched by combining the calculated feature vectors 605, 606, and 607 for each attribute of the search image, but further, an image search method, apparatus, and system according to an embodiment of the present invention In (100), a vector quantization & indexing technique may be applied to search similar images more quickly from a large storage image.

Accordingly, in the image search method, apparatus, and system 100 according to an embodiment of the present invention, after sorting a plurality of elements included in the feature vector for each attribute of the search image by size, the top k having the largest size After calculating an index feature vector having an index value as an element indicating the position in the feature vector for each search image attribute of the elements (608, 609, 610 in FIG. 13), the index feature vector is used to search the image. You can also search for similar images.

For a more specific example, FIGS. 14 and 15 illustrate a method of retrieving a similar image by calculating an index feature vector according to an embodiment of the present invention.

First, as shown in FIG. 14, when two images (both search images and stored images) have similar shapes (for example, clothing A and clothing A'in FIG. 14), the " Shape" criteria (search or store) The feature vector for each image attribute is such that elements with large values are located at similar locations (ie, at similar index values). On the other hand, if the images have different shapes (for example, clothing B in FIG. 14), the feature vectors for each “image” reference image attribute of each image also have significantly different values.

That is, as can be seen in the example of FIG. 14, when the images have similar shapes (clothing A and clothing A'in FIG. 14), feature vectors for each of the “form” reference image attributes for the image are also similar to each other. When an element having a large value is positioned (1001, 1002, 1003 in FIG. 14), and the image has a different shape (for example, clothing B in FIG. 14), the “form” reference image attribute of the image Star feature vectors also have significantly different values. Accordingly, when a feature vector for each "image" reference image attribute of each image shows a similar pattern, the likelihood of each image having the same or similar shape increases.

Therefore, in the present invention, among a plurality of elements included in the feature vector for each image attribute, a predetermined number of elements are calculated based on the size of the elements to construct an index feature vector, and only the partial elements are included. By calculating the similar image using the index feature vector, it is possible to quickly and efficiently search the similar image while reducing the computational resources required.

Further, as shown in FIG. 15, the index feature vector sorts some elements calculated by a predetermined number based on a size in a feature vector for each image attribute including the plurality of elements in order of size thereof (top-k), the index feature vector may be constructed with each index value indicating a position in the feature vector for each image attribute of the calculated some elements.

For a more specific example, when a feature vector 1101 for each image attribute is given in FIG. 15, a predetermined number (8 in FIG. 15) of elements of the feature vector element for each image attribute is given. Calculate ("7, 7, 6, 8, 7, 7, 7, 9" in FIG. 15) and sort by size ("9, 8, 7, 7, 7, 7, 7, in the example above) 6"), the index values (positions) of each index value ("9") indicating the position in the feature vector for each image attribute of the calculated elements are "19", "15", "15", "7 The index feature vector (1102 in FIG. 15) is composed of "2, 3, 16, 17, and 18" (sequentially listed because there are 5), and the index of "6" is "11") ( {19, 15, 2, 3, 16, 17, 18, 11}).

Accordingly, the index feature vector for each image can effectively determine the similarity of each image while having only the element significantly reduced than the feature vector for each image attribute for each image, so that the similar image can be quickly reduced while reducing the computational resources required. And efficient search.

Accordingly, index feature vectors F1, F2, and F3 for the plurality of search images 601, 602, and 603 can be calculated (608, 609, 610), and also stored in a database (611 in FIG. 13). Since the index feature vector for the image is stored, a stored image having an index feature vector similar to the case where the index feature vectors for the plurality of search images 601, 602, and 603 are combined (F1, F2, F3) By searching, it is possible to search for and provide an image similar to an image combining the plurality of search images based on respective search attributes for the plurality of search images 601, 602, and 603.

At this time, when calculating the similarity between the index feature vector of the stored image stored in the database 611 and the index feature vector for the plurality of search images 601, 602, 603 (F1, F2, F3) It is desirable to utilize index-based search techniques such as MTC (Matched Term Count), TF-IDF, and BM25.

16 is a flowchart of an image search method from the perspective of the terminal 110 according to an embodiment of the present invention. More detailed information on the image search method from the perspective of the terminal 110 according to an embodiment of the present invention can be inferred from the description of the image search method from the perspective of the server 120 according to an embodiment of the present invention described above. As it can, a more detailed description is omitted below and the focus is on the subject matter.

First, as can be seen in Figure 16, the terminal 110 transmits a plurality of search data information including one or more search images input by the user to the server 120, the search information transmission step (S210), the server Similar image information receiving step of receiving the similar image information for the searched similar image by combining the search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information from (120) S220) and providing the similar image information providing the similar image information for the similar image to the user (S230).

At this time, in the search information transmission step (S210), a plurality of search attribute information for the plurality of search data information may be transmitted together with the plurality of search data information.

In addition, the search information transmission step (S210), as can be seen in Figure 17, the search image attribute to calculate a feature vector for each of the plurality of search image attributes corresponding to the plurality of search attribute information from the one or more search images It may include a step of calculating a feature vector for each star (S211) and a step of transmitting a feature vector for each search image attribute (S212) for transmitting the calculated feature vectors for each of the search image attributes to the server 120.

Furthermore, in the step of transmitting a feature vector for each of the search image attributes (S211 ), a plurality of search image attribute-specific feature vectors calculated corresponding to the plurality of search attribute information in the one or more search images are connected to each other, thereby generating a single connection search image feature. Vectors can be generated and transmitted to the server 120.

In addition, the computer program according to another aspect of the present invention is characterized in that the computer program stored in a computer-readable medium in order to execute each step of the above-mentioned salpin image retrieval method on a computer. The computer program may be a computer program including high-level language codes that can be executed on a computer using an interpreter, as well as a computer program including machine code generated by a compiler. In this case, the computer is not limited to a personal computer (PC) or a notebook computer, and is equipped with a central processing unit (CPU) such as a server, a smart phone, a tablet PC, a PDA, and a mobile phone, and performs any information processing capable of executing a computer program. Device. In addition, the computer-readable medium may be an electronic recording medium (eg, ROM, flash memory, etc.), magnetic storage medium (eg, floppy disk, hard disk, etc.), optical reading medium (eg, CD-ROM, DVD, etc.).

In addition, FIG. 18 illustrates a configuration diagram of the server 120 that provides image search according to an embodiment of the present invention.

As can be seen in Figure 18, the server 120 that provides an image search according to an embodiment of the present invention includes a search information receiving unit 121, a similar image search unit 122 and a similar image information transmission unit 123 Can be configured.

Below, the server 120 providing an image search according to an embodiment of the present invention is divided into components and examined. For more information about the server 120 that provides an image search according to an embodiment of the present invention can be inferred from the description of the image search method according to an embodiment of the present invention described above, more detailed below Description is omitted.

First, the search information receiving unit 121 receives a plurality of search data information including one or more search images input by a user from the terminal 110.

In this case, the search information receiving unit 121 may receive a plurality of search attribute information for the plurality of search data information together with the plurality of search data information.

In addition, the similar image search unit 122 searches for a similar image by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information.

Finally, the similar image providing unit 123 transmits similar image information for the searched similar image to the terminal.

In addition, FIG. 19 illustrates a configuration diagram of a terminal 110 that provides image search according to an embodiment of the present invention.

As can be seen in Figure 19, the terminal 110 for providing image search according to an embodiment of the present invention includes a search information transmitting unit 111, a similar image information receiving unit 112, and a similar image information providing unit 113. It can be configured to include.

Below, the terminal 110 providing an image search according to an embodiment of the present invention is divided into components and examined. For more information about the terminal 110 providing an image search according to an embodiment of the present invention, it can be inferred from the description of the image search method according to an embodiment of the present invention described above. Description is omitted.

First, the search information transmission unit 111 transmits a plurality of search data information including one or more search images input by a user to the server 120.

In this case, the search information transmitting unit 111 may transmit a plurality of search attribute information for the plurality of search data information together with the plurality of search data information.

In addition, the similar image information receiving unit 112 in the plurality of search data information from the server 120, the search data partial information corresponding to each search attribute information for the plurality of search data information is combined to the searched similar image You will receive similar image information for.

Finally, the similar image information providing unit 113 provides similar image information for the similar image to the user.

Accordingly, in an image search method, apparatus, and computer program according to an embodiment of the present invention, in searching for and providing an image similar to a given search image, a similar image is searched and provided based on specific attributes of a given search image. Further, by searching and providing an image similar to the image combining the plurality of search images based on respective search attributes for the plurality of search images, the similar image for the image combining the plurality of search images is more convenient. And accurately provided.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments described in the present invention are not intended to limit the technical spirit of the present invention, but to explain them, and are not limited to these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical spirits within the equivalent range should be interpreted as being included in the scope of the present invention.

100: image search system
110, 110a, 110b: terminal
111: search information transmitter
112: similar image information receiving unit
113: Similar image information providing unit
120: server
121: search information receiving unit
122: Similar image search unit
123: Similar image information transmitter
124: database
130: network

Claims (15)

A search information receiving step in which the server receives a plurality of search data information including one or more search images input by a user from a terminal;
A similar image search step in which the server searches similar images from a plurality of images stored in a database by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And
A similar image providing step in which the server transmits similar image information on the searched similar image to the terminal;
It includes,
In the database, a predetermined number of elements are calculated based on the size of the elements among a plurality of elements included in the feature vector for each stored image attribute calculated in advance for each of the plurality of images, and then the elements of the some elements are calculated. An index feature vector consisting of index values representing the location is stored,
In the similar image search step,
And searching the similar image using the index feature vector. According to claim 1,
In the step of receiving the search information,
Receiving a plurality of search images input by the user and a plurality of search attribute information for the plurality of search images,
In the similar image search step,
An image retrieval method comprising searching for similar images by combining corresponding retrieval image partial information from the retrieval images for each of the plurality of retrieval attribute information. According to claim 2,
The similar image search step,
A feature vector calculation step for each search image attribute for calculating a feature vector for each search image attribute corresponding to the plurality of search attribute information from the plurality of search images; And
A database search step of searching for similar images using a plurality of stored image attribute-specific feature vectors calculated and stored based on the plurality of search attribute information for a plurality of stored images stored in a database;
Image search method comprising a. According to claim 3,
In the database, a linked storage image feature vector to which the feature vectors for each of the plurality of stored image attributes are connected to the plurality of stored images is stored,
In the similar image search step,
After generating a connection search image feature vector to which the feature vectors for each of the plurality of search image attributes are connected,
And searching for a similar image based on the similarity with the linked storage image feature vector. According to claim 3,
The feature vector calculating step for each attribute of the search image is
Detecting one or more objects included in the search image; And
Calculating a feature vector for each search image attribute based on the search attribute information for each of the one or more objects;
Image search method comprising a. According to claim 3,
In the feature vector calculation step for each attribute of the search image,
After sorting a plurality of elements included in the feature vector for each search image attribute by size, an index feature having as an element an index value indicating a position in the feature vector for each search image attribute of the top k elements having the largest size. An image search method characterized by calculating a vector. According to claim 1,
In the step of receiving the search information,
Receiving search information including a first search image input by the user, first search attribute information therefor, and second search text and second search attribute information therefor;
In the similar image search step,
In the first search image, information obtained by combining first search data part information corresponding to the first search property information and second search data part information corresponding to the second search property information for the second search text is combined. Image search method characterized by searching for similar images using. According to claim 1,
In the step of receiving the search information,
The terminal provides a user interface to select a search image stored in advance by the user and search attribute information therefor.
In the similar image search step,
And searching for a similar image using the previously stored search image and search attribute information for the search image. According to claim 1,
In the step of receiving the search information,
The terminal provides a user interface capable of setting search attribute information for a plurality of search data information input by the user,
Image search method, characterized in that for receiving a plurality of search attribute information set for the plurality of search data information from the terminal. A search information transmission step of transmitting a plurality of search data information including at least one search image input by the user to the server;
Receiving similar image information for similar images searched from a plurality of images stored in a database by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information from the server Receiving similar image information; And
Providing similar image information to provide similar image information for the similar image to the user;
It includes,
In the database, a predetermined number of elements are calculated based on the size of the elements among a plurality of elements included in the feature vector for each stored image attribute calculated in advance for each of the plurality of images, and then the elements of the some elements are calculated. An index feature vector consisting of index values representing the location is stored,
In the step of receiving the similar image information,
The server searches for the similar image using the index feature vector. The method of claim 10,
The step of transmitting the search information,
A feature vector calculation step for each search image attribute for calculating a feature vector for each of the plurality of search image attributes corresponding to the plurality of search attribute information from the one or more search images; And
A step of transmitting a feature vector for each of the search image attributes to transmit the calculated feature vector for each of the search image attributes to the server;
Image search method comprising a. The method of claim 11,
In the step of transmitting a feature vector for each of the search image attributes,
An image retrieval method characterized in that a feature vector for each retrieval image attribute calculated in response to the plurality of retrieval attribute information is connected from the one or more retrieval images to generate one connection retrieval image feature vector and transmit it to the server. . A computer program stored in a computer-readable medium for executing each step of the image retrieval method according to any one of claims 1 to 12 in a computer. A search information receiving unit configured to receive a plurality of search data information including one or more search images input by a user from a terminal;
A similar image search unit for searching similar images from a plurality of images stored in a database by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information; And
A similar image providing unit that transmits similar image information on the searched similar image to the terminal;
It includes,
In the database, a predetermined number of elements are calculated based on the size of the elements among a plurality of elements included in the feature vector for each stored image attribute calculated in advance for each of the plurality of images, and then the elements of the some elements are calculated. An index feature vector consisting of index values representing the location is stored,
In the similar image search unit,
And searching for the similar image using the index feature vector. A search information transmitting unit that transmits a plurality of search data information including one or more search images input by a user to a server;
Receiving similar image information for similar images searched from a plurality of images stored in a database by combining search data partial information corresponding to each search attribute information for the plurality of search data information from the plurality of search data information from the server A similar image information receiving unit; And
A similar image information providing unit that provides similar image information for the similar image to the user;
It includes,
In the database, a predetermined number of elements are calculated based on the size of the elements among a plurality of elements included in the feature vector for each stored image attribute calculated in advance for each of the plurality of images, and then the elements of the some elements are calculated. An index feature vector consisting of index values representing the location is stored,
The server searches for the similar image using the index feature vector.

Images