KR20170026264A - Product search method and system - Google Patents

Abstract

Disclosed is a method and a system for searching a product, by which a product may be recognized from an inquiry image registered by the user and product images including a product that is similar the recognized product to be provided to a user. The method for searching a product according to an embodiment includes: an inquiry image receiving step of, by a communication unit, receiving an inquiry image from a user device; a candidate area extracting step of extracting one or more candidate areas in which a product may be present from the inquiry image by applying an object detection model obtained by learning the inquiry image, by a candidate area detection unit; a feature extracting step of, by the feature extraction unit, applying a candidate image that is an image in the at least one candidate area to the learned feature extraction model to extract features for attributes of the candidate image; and a search step of searching a product image database for product images that are similar to the candidate area image, based on the feature extracted by the search unit. The feature extraction model has a structure in which a low level hierarchy includes a unified network and an upper level hierarchy includes a plurality of classification networks separated for the attributes.

Description

{Product search method and system}

The present invention relates to a product search method and system for recognizing a product from a query image registered by a user and searching for product images including a product similar to the recognized product and providing the product image to a user.

As communication technology developed, various types of Internet shopping malls were created. An Internet shopping mall refers to a business place where goods are traded through the Internet. In order to purchase a product at the Internet shopping mall, the user inputs a text related to the product such as a name of a product or a manufacturer, searches for similar products, selects a predetermined category from the categories provided in the Internet shopping mall, Search.

Thus, conventionally, there is a limit to search for products that are difficult to classify into keywords or categories because they search for products based on text entered by the user or categories selected by the user. This is because, in order to search for a specific object by using a text, the attribute information in which the specific object is classified must be clearly known.

Due to the above problems, it is more convenient for the user to use the image search method of Google (Google.com), for example, when the user searches for the object which does not know the attribute information. Google's image search service is a service model that uses the image itself as a search query and outputs images similar to or related to the image.

Examples of the image retrieval method used in the service model include a method using a tag value of an image and a method using an image hash value. In recent years, a method of searching for similar values using principal component analysis, clustering, and machine learning has also been used in the prior art, in which a pixel value of an image is converted into a matrix for similarity analysis.

However, with regard to fashion merchandise images, such existing image search methods are not effective. This is because the users who are in the process of purchasing the fashion goods do not want to find the images similar to the fashion product images but want to find the fashion products similar or related to the fashion products included in the fashion product images. Accordingly, when the conventional image retrieval method is applied to the fashion product image, since the image retrieval is performed including the pixel information of the unnecessary area such as the background, it is difficult to find the fashion product image desired by the user.

In addition, the user considers a great number of attributes when recognizing fashion goods. Basically, the user considers the color, the brand, and the type of the fashion product when considering the fashion product, and more specifically, takes into consideration the fit, the pattern, the design detail, and the material of the fashion product. As a result, since a fashion product itself has a great number of attributes, when a pixel value of a fashion product image is made into a matrix and is classified using neural network learning, there arises a problem that one fashion product image having various properties must be classified differently . Therefore, there are practical problems in implementing image retrieval as neural network learning.

In addition, when a classification network is learned separately for each attribute of a fashion product and the learned classification network is applied to an image search, a considerable processing time is required.

Korean Patent Registration No. 10-1191172 (entitled "METHOD, APPARATUS AND COMPUTER-READABLE RECORDING MEDIUM FOR MANAGING IMAGES OF IMAGE DATABASE", filed October 9, 2012)

A problem to be solved by the present invention is to learn a neural network in which a lower layer is composed of a unified network and a higher layer is composed of a plurality of classification networks separated by attributes, And to provide a system and method for searching for a product.

A problem to be solved by the present invention is to provide a product search method and system for recognizing a product from a query image registered by a user, searching for product images including a product similar to the recognized product, and providing the product image to the user. More specifically, the present invention provides a product retrieval method and system for estimating a bounding box of a fashion product image so that only attribute information related to fashion is learned and retrieved.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a product search method and system capable of classifying various attribute information of a fashion product and improving the speed of image search without losing the accuracy of the image search.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, there is provided a product retrieval method including: a query image receiving step in which a communication unit receives a query image from a user apparatus; A candidate region extracting step of extracting at least one candidate region in which the product is estimated to exist in the query image by applying the query image to the previously learned object detection model; A feature extraction step of applying a candidate image, which is an image in the at least one candidate region, to a previously-extracted feature extraction model, and extracting features of the candidate image for each attribute; A search step of searching, in a product image database, product images similar to the candidate region image based on the feature extracted by the search unit; Wherein the feature extraction model has a structure in which a lower layer is composed of a unified network and an upper layer is composed of a plurality of classification networks separated by the attribute.

The plurality of classification networks are characterized by coarse features having a resolution lower than a specific resolution and high-resolution features having a resolution higher than the specific resolution for each attribute.

Wherein the feature extraction step comprises: a low-resolution feature extraction step of extracting a low-resolution feature for each attribute by applying the candidate image to the feature extraction model learned as the low-resolution feature; And a high-resolution feature extracting step of extracting a high-resolution feature by the attribute by applying the candidate image to the feature extraction model learned as the high-resolution feature.

Wherein the searching step comprises a low-resolution searching step of searching, in the product image database, product images similar to the candidate image based on the low-resolution feature extracted in the low-resolution feature extracting step; And a search step for searching for product images similar to the candidate image in the similar product images retrieved in the low-resolution retrieval step, based on the high-resolution feature extracted in the high-resolution feature extraction step do.

Other specific details of the invention are included in the detailed description and drawings.

It is possible to recognize a product from a query image registered by a user and to search for product images including a product similar to the recognized product and provide the retrieved product images to the user so that more diverse product images are provided to the user can do.

The retrieval is performed by only the act of registering the query image, thereby making it possible to improve convenience for the user.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a configuration of a product image search system according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of a user apparatus according to an embodiment of the present invention.
3 is a diagram illustrating the configuration of an image search apparatus according to an embodiment of the present invention.
4 is a diagram illustrating the structure of Deep Neural Networks (DNN).
5 is a diagram illustrating the structure of Convolutional Deep Neural Networks (CNN).
FIG. 6 is a diagram illustrating a process of calculating a product product.
7 is a diagram illustrating a sub-sampling process.
8 is a diagram illustrating product images retrieved based on a query image and a query image.
FIG. 9 is a flowchart illustrating a product image search method according to an embodiment of the present invention.
10 is a diagram illustrating a configuration of an image search apparatus according to another embodiment of the present invention.
11 is a diagram for explaining a learning process of the object detection model used by the candidate region extraction unit of FIG.
12 is a diagram showing a structure of a feature extraction model used by the feature extraction unit of FIG.
13 is a diagram for explaining the operation of the search unit of FIG.
14 is a diagram illustrating product images retrieved based on features of candidate regions of a query image.
15 is a diagram illustrating product images retrieved based on features of a candidate region selected from candidate regions of the query image.
16 is a flowchart illustrating a product image search method according to another embodiment of the present invention.
FIG. 17 is a graph showing the results of an experiment on the time required for feature extraction and transmission with respect to feature extraction models having different structures.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element. Like reference numerals refer to like elements throughout the specification and "and / or" include each and every combination of one or more of the elements mentioned. Although "first "," second "and the like are used to describe various components, it is needless to say that these components are not limited by these terms. These terms are used only to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense that is commonly understood by one of ordinary skill in the art to which this invention belongs. In addition, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a configuration of a product image search system 1 according to an embodiment of the present invention.

Referring to FIG. 1, a merchandise image search system 1 according to an embodiment of the present invention includes a user apparatus 100 and an image search apparatus 200.

The user device 100 receives a query image from a user. The query image can be a two-dimensional color image including a fashion item (hereinafter, referred to as a 'product') such as clothing, shoes, bags and accessories. This query image may be either distributed to the user device 100 from another device (not shown), acquired through a camera (not shown) included in the user device 100, It may have been obtained through a screen capture function.

The user apparatus 100 transmits a query image input by the user to the image search apparatus 200 through the wired or wireless network 300 and transmits a query including a product similar to the product included in the query image Images are provided. The product images provided by the image search apparatus 200 may be sorted according to a predetermined reference and output through the user apparatus 100. [ For example, the product images can be sorted based on the degree of similarity with the products included in the query image. Such a user device 100 may comprise a wired or wireless communication device. Examples of wired / wireless communication devices include personal computers (PCs), smart phones, and tablet PCs. A more detailed description of the configuration of the user device 100 will be given later with reference to Fig.

The image search apparatus 200 provides a product image search service to the user apparatus 100. Specifically, the image search apparatus 200 receives a query image from the user device 100, recognizes the product from the received query image (label prediction), and extracts features of the recognized product. According to an embodiment, the image search apparatus 200 recognizes a product from a query image based on Deep Learning and extracts features of the recognized product. Then, the image search apparatus 200 searches the product image database (refer to '280' in FIG. 3) using the recognized product and the extracted feature, and the product image including the product similar to the product recognized in the query image . And transmits the retrieved product images to the user device 100. A more detailed description of the configuration of the image search apparatus 200 will be described later with reference to FIG.

2 is a diagram illustrating a configuration of a user apparatus 100 according to an embodiment of the present invention.

2, the user device 100 includes an input unit 110, an output unit 1120, a communication unit 130, a storage unit 140, a power unit 150, and a controller 160.

The input unit 110 receives commands and information from a user. For example, the input unit 110 receives a selection command for selecting a query image. For this, the input unit 110 may include input means including a touch pad, a key pad, a button, a switch, a jog wheel, or a combination thereof. The touch pad may be stacked on a display (not shown) of the output unit 120 to be described later to form a touch screen.

The output unit 120 outputs the command processing result and various information to the user. For example, the output unit 120 outputs product images received from the server 200. To this end, the output unit 120 may include a display and a speaker, although not shown in the figure. The display may be provided in any form well known in the art, such as a flat panel display, a flexible display, an opaque display, a transparent display, an electronic paper (E-paper) . The output unit 120 may further include any type of output means well known in the art to which the present invention belongs, in addition to the display and the speaker.

The communication unit 130 communicates with the image search apparatus 200 through the wired / wireless network 300. To this end, the communication unit 130 supports a wired communication method and / or a wireless communication method supporting the TCP / IP protocol or the UDP protocol. Examples of wireless communication methods include Wireless Broadband Internet, WiFi, ZigBee, Bluetooth (e.g., Bluetooth 4.0), Ultra Wide Band (UWB) Field Communication (NFC), Third Generation Mobile Communication (3G), Fourth Generation Mobile Communication (4G), and Fifth Generation Mobile Communication (5G).

The storage unit 140 stores data, programs, and applications necessary for the user device 100 to operate. Such storage 140 may be any type of computer readable medium such as non-volatile memory, volatile memory, built-in memory, removable external memory, hard disk, optical disk, magneto-optical disk, or any form of computer readable storage medium known in the art And the like. Examples of the external memory include an SD card (Secure Digital card), a mini SD card, and a micro SD card.

The power supply unit 150 supplies power to the respective components of the user device 100. According to one embodiment, the power supply 150 may be implemented to be mechanically and electrically disconnectable from the user device 100. The separated power source unit 150 can be replaced with an extra power source unit (not shown). According to another embodiment, the power supply unit 150 may be implemented integrally with the user device 100. [ In this case, the power supply unit 150 can be charged with electric power supplied from a separately provided charging device (not shown). At this time, the power supply unit 150 can receive power from the charging device according to the wired power transmission technique or the wireless power transmission technique.

The control unit 160 connects and controls the other components in the user device 100. [ For example, the control unit 160 configures a user interface for displaying a query image and / or a product image, and displays the user interface through a display.

FIG. 3 is a diagram illustrating a configuration of an image search apparatus 200 according to an embodiment of the present invention.

3, an image search apparatus 200 according to an exemplary embodiment of the present invention includes a communication unit 210, a feature extraction unit 230, a search unit 240, a similarity calculation unit 250, 260, a query image database 270, and a merchandise image database 280.

The communication unit 210 is in charge of communication with the user device 100. For example, the communication unit 210 receives a query image (refer to '310' in FIG. 8) from the user device 100. In another example, the communication unit 210 transmits retrieved product images (see 320 in FIG. 8) to the user device 100 based on the query image. To this end, the communication unit 210 may support a wired communication method and / or a wireless communication method.

The feature extraction unit 230 extracts features of the query image. According to the embodiment, the feature extracting unit 230 extracts the feature of the query image through the learned model based on the Deep Learning.

Deep learning is a set of machine learning algorithms that try to achieve a high level of abstraction (a task that summarizes key content or functions in large amounts of data or complex data) through a combination of several nonlinear transformation techniques. Is defined. Deep learning can be viewed as a field of machine learning that teaches computers how people think in a big way.

When there is any data, it is represented by the form that the computer understands (for example, the pixel information is represented by a column vector in the case of the image), and many researches How to create expression models and how to model them). As a result of these efforts, various deep-running techniques have been developed. Deep learning techniques include Deep Neural Networks (DNN), Convolutional Deep Neural Networks (CNN), Recurrent Neural Networks (RNN), and Deep Belief Networks (DBN). have.

Deep Neural Networks (DNN) is an Artificial Neural Network (ANN) composed of hidden layers between an input layer and an output layer.

4 is a diagram illustrating a structure of a depth-of-field neural network. In FIG. 4, each circle represents one perceptron. A perceptron consists of several input values, a processor, and a single output value. The processor multiplies the multiple input values by their respective weights, and then weights all the input values multiplied by the weights. The processor then substitutes the summed value into the activation function and outputs one output value. If you want to output a specific value as the output value of the activation function, you can modify the weight multiplied by each input value, and recalculate the output value using the modified weight. In Fig. 4, each perceptron may use a different activation function. Each perceptron also receives the outputs from the previous layer as input, and then uses the activation function to obtain the output. The obtained output is transferred to the input of the next layer. After the process described above, several output values can be finally obtained.

Going back to the description of the deep running technique, Convolutional Deep Neural Networks (CNN) is a kind of multilayer perceptrons designed to use minimal preprocessing. The composite neural network consists of one or several convolutional layers and general artificial neural network layers stacked on top of it, and utilizes additional weighting and pooling layers. This structure allows the composite neural network to fully utilize the input data of the two-dimensional structure. In addition, the combined product neural network can be trained through standard inverse delivery. Composite neural networks are more easily trained than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

The composite neural network extracts features from the input image by alternately performing the composite product and the subsampling on the input image. FIG. 5 is a diagram illustrating a structure of a composite neural network. Referring to FIG. 5, the composite neural network includes a plurality of convolution layers, a plurality of subsampling layers (Max-Pooling layer, ), And a fully-connected layer. The composite product layer is a layer that performs a composite product on an input image. The subsampling layer is a layer for extracting a maximum value locally for an input image and mapping it to a two-dimensional image.

In the composite product layer, information such as the kernel size, the number of kernels to be used (that is, the number of maps to be generated), and a weight table to be used in the composite product operation are required. For example, consider the case where the size of the input image is 32 × 32, the size of the kernel is 5 × 5, and the number of kernels to be used is 20. In this case, if a 5 × 5 kernel is applied to a 32 × 32 input image, it is impossible to apply the kernel to two pixels on the upper, lower, left, and right sides of the input image. 6, when the kernel is placed on the input image and the resultant product is performed, the resulting value '-8' is the number of pixels in the kernel of the input image included in the kernel, Is determined by the value of the pixel corresponding to the center element of the pixel. Therefore, if a 5 × 5 kernel is applied to a 32 × 32 input image and a composite product is performed, a 28 × 28 map is generated. It is assumed that the total number of kernels to be used is 20 in total. Therefore, a total of 20 28 × 28 maps are generated in the first convolution layer (see 'C1-layer' in FIG. 5).

At the subsampling layer, information on the size of the kernel to be sub-sampled, information on whether to select the maximum value or the minimum value among the values in the kernel area is required. 7 is a diagram illustrating a sub-sampling process. Referring to FIG. 7, it can be seen that the size of the kernel to be sub-sampled is 2 × 2, and the maximum value among the values included in the kernel area is set to be selected. When a 2 × 2 kernel is applied to an 8 × 8 size input image, a 4 × 4 size output image can be obtained. That is, an output image whose size is reduced to 1/2 of the input image can be obtained.

Returning to the description of the deep running technique, the Reccurent Neural Network (RNN) refers to a neural network in which the connections between the units forming the artificial neural network constitute a directed cycle. Circular neural networks can utilize memory inside a neural network to process arbitrary inputs, unlike pre-feed neural networks.

Deep Belief Networks (DBN) is a generative graphical model used in machine learning. In deep learning, it is a deep neural network composed of multiple layers of latent variables. There is a connection between layers but there is no connection between the units in the layer.

The deep trust neural network can be used for preliminary learning due to the nature of generation model. After learning initial weight through preliminary learning, we can fine - tune the weight through back propagation or other discrimination algorithm. This characteristic is very useful when the training data is small, because the smaller the training data is, the more the initial value of the weight affects the resulting model. The initial value of the weighted initial value is closer to the optimal weight value than the arbitrary initial value of the weighted value, which enables improvement of the performance and speed of the unadjusted step.

Referring again to FIG. 3, the feature extraction unit 230 can extract the feature of the query image using the model learned by the composite neural network technique among the deep learning techniques as described above. Then, the feature extraction unit 230 recognizes the product in the query image based on the extracted feature. For example, it recognizes the category of goods (bag, shoes, shirt, pants) in the query image. The recognition result for the category of the goods is provided to the search unit 240, which will be described later, and the features extracted from the query image are provided to the similarity calculation unit 240, which will be described later.

In addition, the feature extraction unit 230 may extract the feature of the product images stored in the product image database 270. [

According to one embodiment, the feature extraction operation for the product images can be performed only once at a time. In this case, the features extracted from each product image can be mapped and stored in each product image.

According to another embodiment, the feature extraction operation for the product images may be performed each time the feature extraction operation for the query image is completed. In addition, the product images stored in the product image database may be periodically updated, and feature extraction operations for the product images may be performed each time a new product image is stored in the product image database.

The search unit 240 searches the goods image database 280 for goods images including goods similar to the goods recognized in the query image. Specifically, the search unit 240 searches the product image database 280 for product images that include products corresponding to or similar to the category of the product recognized in the query image. The product images retrieved by the retrieval unit 240 are provided to the similarity calculation unit 240 to be described later.

The similarity calculation unit 250 calculates the similarity between the product images retrieved by the retrieval unit 240 and the query image. Specifically, the similarity calculation unit 250 calculates the similarity between the feature extracted from the query image and the feature of each retrieved product image. In this case, the similarity calculation method may be exemplified by Hamming distance and cosine similarity. The similarity value calculated for each product image is provided to the image arranging unit 260 to be described later.

The image arranging unit 260 arranges the product images retrieved by the retrieving unit 240 based on the similarity value provided from the similarity calculating unit 250. For example, the image arranging unit 260 arranges product images in the order of high similarity value. The sorted merchandise images are transmitted to the user apparatus 100 through the communication unit 210.

According to one embodiment, all of the ordered merchandise images are provided to the user device 100.

According to another embodiment, among the sorted product images, product images whose similarity value is equal to or higher than the reference value are selected, and only the selected product images are transmitted to the user device 100.

According to another embodiment, the top N product images are selected from among the sorted product images, and only the selected product images are transmitted to the user device 100. In one example, N may be preset by the administrator (not shown) of the server 200, and the set values may be changed by the user via the user device 100. [ As another example, N may be automatically determined according to the purchase frequency or purchasing history of the user. For example, N may be set to a small value for a user with a low purchase frequency, and N may be set to a large value for a user with a high purchase frequency.

The query image database 270 stores the query image received from the user device 100. [ The query image can be classified and stored for each user. Query images classified by users can be used to analyze the type of products preferred by users.

The merchandise image database 280 stores merchandise images. For example, the merchandise image may be uploaded by the administrator of the server 200. [ As another example, the merchandise image may be automatically collected from another apparatus (not shown) or another server (not shown) associated with the server 200.

The feature is extracted for the uploaded or collected product image, and the category of the product in the product image is recognized based on the extracted feature. The feature extracted from the product image can be mapped to the product image, and the product image having the feature mapped can be classified based on the recognized category and stored in the product image database 280. [ Here, the category of the product image includes clothes, shoes, bags, accessories, and the like. The categories as illustrated can be further subdivided into subcategories. For example, the category 'apparel' may be subdivided into subcategories such as shirts, skirts, pants, dresses, and so on. And the category 'shoes' can be further subdivided into subcategories such as sneakers, shoes, boots, slippers, and so on.

The server 20 according to the embodiment of the present invention has been described above with reference to FIGS. In the above example, the server 200 includes the feature extraction unit 230, the similarity calculation unit 250, and the image alignment unit 260. However, one or more of the components Or may be provided in the user device 100.

In addition, the server 200 may further include one or more other components in addition to the components shown in FIG. For example, the server 200 may include an image collecting unit for collecting product images from other external servers linked to the server 200, a user information database unit for storing personal information or purchasing history of users, An analysis unit for analyzing the history, and a condition setting unit for automatically setting various conditions related to the product image search service based on the analysis result of the analysis unit.

FIG. 9 is a diagram illustrating a fashion product image searching method according to an embodiment of the present invention.

First, the server 200 receives a query image from the user device 100 (S410).

Thereafter, the server 200 extracts features from the query image using the model learned by deep learning (S420). Here, the model may be a model learned by the composite neural network technique. However, the model is not necessarily limited to a model learned by the composite neural network technique, and it is needless to say that a model learned by other types of deep learning techniques may be used.

Thereafter, the server 200 recognizes the goods in the query image based on the features extracted from the query image (S430). Specifically, the server 200 recognizes the category of the product in the query image based on the feature extracted from the query image.

Then, the server 200 searches the goods image database 270 for goods images including goods similar to the goods recognized in the query image (S440). Specifically, the server 200 searches the product image database 270 for product images that are similar to or similar to the category of the product recognized in the query image.

Thereafter, the server 200 calculates the degree of similarity between the product images retrieved from the product image database 270 and the query image (S450). The step S450 may include calculating the similarity between the feature extracted from the query image and the feature extracted from each product image. As a method for calculating the degree of similarity, the Hamming distance and the degree of cosine similarity can be exemplified.

Thereafter, the server 200 arranges the product images based on the similarity value calculated for each product image (S460). In operation S460, the merchandise images may be sorted in descending order of the similarity value.

Thereafter, the server 200 selects a predetermined number of commodity images from the sorted commodity images (S470). According to an exemplary embodiment, the step S470 includes a step of selecting product images whose similarity value is equal to or higher than a reference value. According to another embodiment, the step S470 includes selecting the top N product images based on the similarity value. At this time, the N may be set by the administrator or user of the server 200, or may be automatically determined in proportion to the purchase frequency of the user.

The selected product images are transmitted to the user device 100 through the wired / wireless network 300 (S470). The transmitted product images are displayed on the display of the user device 100 together with the query image. For example, the query image 310 and the retrieved product images 320 as shown in FIG. 8 are displayed through the display.

As described above, the fashion product image searching method according to the embodiment of the present invention has been described with reference to FIG. Some of the steps shown in FIG. 9 may be omitted. For example, step S470 may be omitted.

FIG. 10 is a diagram illustrating a configuration of an image search apparatus 500 according to another embodiment of the present invention.

10, an image search apparatus 500 according to another embodiment of the present invention includes a communication unit 510, a candidate region extraction unit 520, a feature extraction unit 530, a search unit 540, An image arranging unit 560, a query image database 570, and a merchandise image database 580. The components of the image search apparatus 500 shown in FIG. 10 are the same as or similar to those of the server 200 shown in FIG. 3, so that redundant description will be omitted and differences will be mainly described.

The candidate region extraction unit 520 extracts one or more candidate regions from the query image.

According to one embodiment, when the user performs a touch operation or the like to designate a predetermined region in the query image, the candidate region extraction unit 520 extracts the designated region as a candidate region.

According to another embodiment, the candidate region extraction unit 520 automatically extracts candidate regions in which a product is estimated to exist in the query image. For this, the candidate region extraction unit 520 extracts candidate regions from the query image using an object detection model. Examples of object detection models include R-CNN (Regions with Convolutional Neural Networks, Region based Convolutional Neural Networks), fast R-CNN models, and YOLO (You Only Look Once) models.

The object detection model 521 used by the candidate region extraction unit 520 can be learned in advance by using the ground truth image of the input image and the input image. Hereinafter, the learning process of the object detection model 521 used by the candidate region extraction unit 520 will be described in more detail with reference to FIG.

Referring to FIG. 11, the object detection model 521 generates a plurality of bounding boxes in an area where the goods are estimated to exist in the input image 810. FIG. 11 shows a case where a total of four bounding boxes are generated in the input image 820. FIG.

Thereafter, the object detection model 521 compares the input image 820 in which the bounding box is created with the ground-truth image 830 of the input image 810. The ground-truth image 830 refers to an image in which a bounding box is generated for each product, and includes position information of each bounding box and product information of the products in each bounding box. The product information includes attribute information of the product. The attribute information of a product includes an item type, a design detail, a pattern, and a material.

The object detection model 521 regenerates a plurality of bounding boxes in the input image 840 with reference to information (for example, position and product) of the plurality of bounding boxes generated in the ground-truth image 830 . 11 shows a case where a total of three bounding boxes are regenerated in the input image 840. FIG. When the input image 820 and the input image 840 are compared with each other, the positions of the plurality of bounding boxes generated in the input image 840, as compared with the input image 820, You can see that this is similar to the position of the bounding box.

Referring again to FIG. 10, the candidate region extraction unit 520 extracts candidate regions from the query image using the object detection model 521 that has been sufficiently learned in the above-described manner. A process of extracting a candidate region from a query image will be described in detail as follows.

First, the candidate region extraction unit 520 extracts a feature map that is an intermediate step of the learned object detection model 521. Then, a weighted sum of feature maps having a large influence on the classification is calculated, and a position of an image to be used mainly is generated as a heat map. Then, a bounding box is created by applying post processing such as thresholding or connected component extraction to the generated heat map.

If there are several kinds of items in the query image, a bounding box is created for each item. The object detection model 521 includes not only classification models for classifying the types of items but also classification models for classifying types of design details, classification models for classifying types of patterns, classification classes for classifying materials Using the model, it is possible to estimate the approximate position of a feature on the query detail, pattern, or material on the query image.

The feature extraction unit 530 extracts features of each of the plurality of candidate regions extracted from the query image. More specifically, the feature extraction unit 530 extracts a coarse feature and a fine feature for each of a plurality of candidate regions extracted from the query image. For this, the feature extraction unit 530 uses a feature extraction model. Here, the structure of the feature extraction model will be described in detail with reference to FIG.

Referring to FIG. 12, the feature extraction model is composed of a single network 531 and a plurality of classification networks 532 separated by attributes.

The single network 531 represents a plurality of convolutional layers (C1-layer to C3-layer) and multiple subsampling layers (MP1-layer to MP2-layer) in the composite neural network shown in FIG. If the lower layer is configured as a unified network, the lower layer can share characteristics of different attributes.

The plurality of classification networks 532 represent a fully-connected layer in the composite neural network shown in FIG. According to an embodiment, the plurality of classification networks 532 includes an item classification network, a design detail classification network, a pattern classification network, and a material classification network. The item classification network extracts a coarse feature for an item and then extracts a high-resolution feature. The design detail classification network extracts the low-resolution features for the design details, and then extracts the high-resolution features. The pattern classification network extracts low-resolution features for the pattern, and then high-resolution features. The material classification network then extracts the low-resolution features of the material and then the high-resolution features.

Referring again to FIG. 10, the feature extraction unit 530 recognizes at least one item, design detail, fashion, and material of a product in each candidate region based on the feature extracted for each attribute in each of the plurality of candidate regions.

The candidate region extracting unit 520 and the feature extracting unit 530 may operate in the same manner with respect to the product image stored in the product image database 580. [ That is, the candidate region extracting unit 520 extracts a plurality of candidate regions from the product image. The feature extraction unit 530 extracts a coarse feature and a fine feature for each of the plurality of candidate regions extracted from the product image. As described above, the plurality of candidate regions extracted from the product image, the low-resolution feature and the high-resolution feature extracted for each attribute in each candidate region can be mapped to the corresponding product image and stored in the product image database 580.

The search unit 540 searches the product image database 280 for product images including products similar to items (design details, patterns, or materials) of goods recognized in each candidate region of the query image. Hereinafter, the operation of the search unit 540 will be described in more detail with reference to FIG.

A candidate region including a T-shirt is extracted from the query image shown in FIG. 13, and an image in the extracted candidate region (hereinafter, referred to as a 'candidate image') is applied to the feature extraction model to obtain a low- coarse feature and a fine feature are extracted.

Then, the search unit 540 searches for product images similar to the candidate image using a binary value of the low-resolution feature of the item. For example, when the binary value for the low-resolution feature of the item is '10100', the search unit 540 adds '10100' to the item hash table among the product images stored in the product image database 580 (For example, a value with a Hamming distance of N or less, and N may vary depending on the type and amount of the product image stored in the product image database), or a value similar to '10100' Search for images. Here, the item hash table refers to a table in which characteristics of an item are stored in association with a predetermined product image.

Then, the search unit 540 uses the binary value of the high-resolution feature of the item to search again for the product images similar to the candidate image within the product images retrieved using the low-resolution feature of the item. As described above, the search unit searches the product images similar to the candidate image by using the low-resolution feature of the item, and then secondarily searches the product images similar to the candidate image within the first-retrieved product images using the high-resolution feature of the item .

Alternatively, although not shown in FIG. 13, according to another embodiment, the search unit 540 may perform a binary search for a high-resolution feature of the design detail, a binary value for the high-resolution feature of the pattern, A binary value for the feature may be used. In this case, the retrieval unit 540 may refer to a design detail hash table, a pattern hash table, or a material hash table instead of the item hash table of previously stored product images.

As described above, if the product images similar to the candidate image are firstly searched based on the low resolution of the item, the search range can be limited, so that the search speed can be increased. In addition, when the product images similar to the candidate image are searched for in the secondary search, the product images can be searched based on the high-resolution features of the item, design detail, pattern, or material, The probability that a user provides a desired search result is higher than in a case where a secondary search is performed using only the feature.

Meanwhile, in the product image search process as described above, the user may not intervene or the user may intervene. 14 and 15 will be referred to for a specific explanation.

14 is a diagram for explaining a case where the user does not intervene in the product image search process. And FIG. 15 is a diagram for explaining a case where a user intervenes in a product image search process.

If the first candidate region 631, the second candidate region 632, the third candidate region 633 and the fourth candidate region 634 are extracted from the query image 630 shown in FIG. 14, the search unit 540 Retrieves product images 640 that include items similar to the item, design detail, material, or pattern of the goods recognized in each of the candidate regions 631, 632, 633, and 634.

More specifically, the search unit 540 may include a product similar to or identical to an item of merchandise (e.g., sunglasses), design detail, pattern or material recognized in the first candidate region 631 of the query image 630 Product images 641 are retrieved. The retrieval unit 540 may also include a product similar to or identical to an item of merchandise (e.g., a striped t-shirt), design detail, pattern or material recognized in the second candidate region 632 of the query image 630 Product images 642 are retrieved. The retrieval unit 540 may also include a product similar to or identical to an item of merchandise (e.g., skinny), design detail, pattern or material recognized in the third candidate region 633 of the query image 630 Product images 643 are retrieved. The search unit 540 may also search for a product containing items similar or identical to items of merchandise (e.g., boots), design details, patterns or materials recognized in the fourth candidate region 634 of the query image 630 Images 644 are retrieved.

Before the retrieval unit 540 retrieves the product images based on the attributes of the goods recognized in the respective candidate regions of the query image 630, the candidate region extraction result for the query image 630 is transmitted to the user device 100 And may be displayed via the user device 100. [ Then, when the user selects a predetermined candidate region, the information on the selected candidate region is transmitted to the image search apparatus 500, and the search unit 540 of the image search apparatus 500 searches the candidate region The merchandise images including merchandise that is similar or identical to merchandise merchandise. For example, if the second candidate region 632 is selected by the user from the query image 630 and the candidate regions 631, 632, 633, and 634 shown in FIG. 15, the search unit 540 searches the second candidate region 632 Only product images 642 that are similar or identical to item items (e.g., striped t-shirts), design details, patterns, or materials recognized in the candidate region 632 are searched.

At this time, the retrieved merchandise images 642 may include merchandise images 642a, 642b, and 642c including only striped t-shirts and / or merchandise images 642d including persons wearing striped t-shirts. If you provide product images that include people wearing striped T-shirts as well as product videos that include only striped T-shirts, you can see the entire style associated with striped T-shirts. Therefore, the user does not have to search for the styling method related to the striped T-shirt separately, thereby improving the user's convenience.

Referring again to FIG. 10, the product images retrieved from the retrieval unit 540 are provided to the similarity calculation unit 550, which will be described later.

The similarity calculation unit 550 calculates the degree of similarity between the product images retrieved by the retrieval unit 540 and the query image. For example, the similarity calculation unit 550 calculates the similarity between features extracted from each candidate region of the query image and features extracted from each candidate region of the product image. In this case, the similarity calculation method may be exemplified by Hamming distance and cosine similarity.

The image arrangement unit 560 arranges the product images retrieved by the retrieval unit 540 based on the similarity value provided from the similarity calculation unit 550. For example, the image arranging unit 560 arranges product images in the order of high similarity value. The sorted commodity images are provided to the user apparatus 100 through the communication unit 510.

At this time, all the ordered product images are transmitted to the user device 100, or only the product images selected from the sorted product images are transmitted to the user device 100. For example, among the sorted product images, only the product images having a similarity value equal to or higher than the reference value may be selected, or the uppermost M product images among the sorted product images may be selected. At this time, M may be manually set by an administrator or a user of the image search apparatus 500, or may be automatically set according to a purchase frequency or purchasing history of the user.

FIG. 16 is a flowchart illustrating a fashion product image searching method according to another embodiment of the present invention.

First, the image search apparatus 500 receives a query image from the user apparatus 100 (S710).

Thereafter, the image search apparatus 500 applies the query image to the object detection model 521 learned by deep learning to extract one or more candidate regions estimated to be present in the query image (S720). Here, the object detection model may be an R-CNN model, a fast R-CNN model, or a YOLO model. The illustrated models can be learned using learning data including an input image and a ground-truth image of the input image. Or may be learned using learning data including an input image.

Then, the image search apparatus 500 applies a candidate image, which is an image in the candidate region, to the feature extraction model learned by deep learning, and extracts features for each attribute of the candidate region (S730)

According to an embodiment of the present invention, the step S730 may include a step of applying a candidate image to the learned feature extraction model to extract a coarse feature for each attribute of the candidate image, and a step of applying the candidate image to the feature extraction model And extracting a fine feature for each attribute of the candidate image.

As described above with reference to FIG. 12, the feature extraction model has a structure in which the lower layer is composed of a unified network and the upper layer is composed of a plurality of classification networks. If the structure of the feature extraction model is constructed as described above, the lower layer is composed of a single network, so that parameters can be shared in the lower layer. Also, since only the upper layer is divided into a plurality of classification networks, it is possible to output low-resolution features and high-resolution features for each item, design detail, pattern, and material. In addition, the average time required for feature extraction can be shortened, and the speed of product image retrieval can be improved. The video search speed improvement will be described later with reference to FIG.

Thereafter, the image search apparatus 500 searches the product image database 580 for product images similar to the candidate image, based on the features extracted from the candidate images (S740).

According to the embodiment, the step S740 may include a step of primarily searching for a product image similar to the candidate image in the product image database 580 based on the low-resolution feature extracted from the candidate image, And secondarily searching for the commodity images similar to the candidate image in the commodity images retrieved first.

In the first search step using the low-resolution feature of the candidate image, product images that are approximately similar to the candidate image are searched. In the second search step using the high resolution feature of the candidate image, product images more similar to the candidate image are searched.

On the other hand, the low-resolution feature of the candidate image item can be used in the first search step. In addition, not only the high-resolution features of the items of the candidate image but also the high-resolution features of the design details of the candidate images, the high-resolution features of the materials, the high-resolution features of the patterns, or the high-resolution features of the materials may be used in the secondary search step. It is also possible for the user to select which attribute of the high resolution feature to use in the secondary search step.

Then, the image search apparatus 500 calculates the similarity between the retrieved product images and the query image (S750). The step S750 includes calculating the similarity between the feature extracted from each candidate region of the query image and the feature extracted from each candidate region of the product image.

Thereafter, the image search apparatus 500 arranges the product images based on the calculated similarity value (S760). The step S760 may include arranging the product images in descending order of the similarity value.

Then, the image search apparatus 500 selects a predetermined number of product images from among the sorted product images, and transmits the selected product images to the user device (S770).

According to one embodiment, in step S770, product images having a similarity value equal to or higher than a reference value may be selected.

According to another embodiment, in step S770, the top M product images may be selected based on the similarity value. At this time, M may be set by an administrator or a user of the image search apparatus 500, or may be automatically determined in proportion to the purchase frequency of the user.

FIG. 17 is a graph showing the results of an experiment on the time required for feature extraction and transmission with respect to feature extraction models having different structures.

The objects to be tested are the first feature extraction model, the second feature extraction model, and the third feature extraction model. The first feature extraction model consists of one single classification network. The second feature extraction model is composed of a single network and a plurality of classification networks (items, materials, patterns). The third feature extraction model is composed of a plurality of single classification networks (items, materials, patterns). Here, the second feature extraction model means the feature extraction model shown in FIG.

10,000 test images were used in the experiment. Experimental results show that the average time taken for the feature extraction forward in the first feature extraction model is measured to be 18 ms. The mean time required for feature extraction in the second feature extraction model was measured to be 27 ms. The average time required for feature extraction and transmission in the third feature extraction model was measured to be 54 ms.

That is, it can be seen that the speed of the second feature extraction model is improved by about two times as compared with the third feature extraction model. Therefore, it is confirmed that the lower layer is composed of a single network, and the upper layer is useful for improving the image retrieval speed when a second feature extraction model composed of a plurality of classification networks (item, material, pattern) is used have.

The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in software modules executed in hardware, or in a combination of both. The software module may be a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD- May reside in any form of computer readable recording medium known in the art to which the invention pertains.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

100: User device
200: Image search device
210:
230: Feature extraction unit
240:
250:
260:
270: Query image database
280: Product image database

Claims (4)

A communication unit receiving a query image from a user device;
A candidate region extracting step of extracting at least one candidate region in which the product is estimated to exist in the query image by applying the query image to the previously learned object detection model;
A feature extraction step of applying a candidate image, which is an image in the at least one candidate region, to a previously-extracted feature extraction model, and extracting features of the candidate image for each attribute;
A search step of searching, in a product image database, product images similar to the candidate region image based on the feature extracted by the search unit;
Lt; / RTI >
Wherein the feature extraction model has a structure in which the lower layer is composed of a unified network and the upper layer is composed of a plurality of classification networks separated by the attribute. The method according to claim 1,
Wherein the plurality of classification networks comprise:
Wherein a coarse feature having a resolution lower than a specific resolution and a fine feature having a resolution higher than the specific resolution are learned for each attribute. 3. The method of claim 2,
The feature extraction step may include:
Extracting a low-resolution feature for each of the plurality of attributes by applying the candidate image to the learned feature extraction model with the low-resolution feature; And
A high-resolution feature extraction step of extracting a high-resolution feature for each of the attributes by applying the candidate image to the feature extraction model learned as the high-resolution feature;
And displaying the product image. The method of claim 3,
The retrieving step comprises:
A low resolution searching step of searching, in the product image database, product images similar to the candidate image based on the low resolution feature extracted in the low resolution feature extraction step; And
Searching for a product image similar to the candidate image within the similar product images retrieved in the low-resolution retrieval step, based on the high-resolution feature extracted in the high-resolution feature extraction step;
And displaying the product image.

Images