KR20190117958A - Method for image searching based on combination of picture and sketch - Google Patents

Abstract

Disclosed in the present invention is a method for searching a similar image by utilizing photo sketch combination. The similar image searching method executed by a computer comprises: a step of searching and providing a clothing photo corresponding to a target image selected by a user from a clothing database which stores a plurality of clothing photos different from one another by category and sketch images related to the plurality of clothing photos; a step of receiving input of a sketch to add details in the clothing photo for the provided clothing photos; a step of extracting integrated clothing feature representation from an integrated image including the sketch drawn on the provided clothing photo and the provided clothing photo; and a step of searching and providing an image having the highest similarity to the integrated image based on the integrated clothing feature representation for the clothing database. Therefore, the method for searching a similar image by utilizing photo sketch combination can search and provide an image similar to a combination image of a photo and a sketch based on a learning model (CNN model) learned through deep learning.

Description

{METHOD FOR IMAGE SEARCHING BASED ON COMBINATION OF PICTURE AND SKETCH}

Embodiments of the present invention relate to a technique for searching for similar images, and more particularly, to a technique for searching for and providing a similar image by combining a photograph and a sketch.

Smart devices such as smartphones and tablets are portable by the user and can access contents anytime and anywhere through the wireless Internet. Such high accessibility and the use of various contents have made users habitually use smart devices and use various functions through smart devices. For example, you can search for news, weather, and more before you start your day, search for shopping, hairstyles, clothing, etc. on your commute or subway, or search for restaurants and appointments.

Interest in styles such as fashion and makeup is increasing regardless of gender, age, and age, and as the importance and interest of fashion styling increases, fashion style services through smart devices and the Internet are provided.

Korean Patent Laid-Open No. 10-2017-0096971 relates to a product recommendation method using a style feature, which obtains a query image from a user client, extracts a category and style feature of the image, and learns using a plurality of learning data images. A technique for retrieving a merchandise image having a style feature similar to that of a query image from a merchandise image database is disclosed using the model.

The present invention combines a sketch drawn by a user using a touch tool (for example, a touch fan, a hand, etc.) on a garment photo and a photo, based on a learning model (CNN model) trained through deep learning. The present invention relates to a technique for searching for and providing an image similar to a combined image of a sketch.

In addition, by repeatedly searching for an image similar to a single image including a sketch, a sketch and a photo on the similar image provided through the search until the image desired by the user is provided (that is, performing a refinement), Even though the image selected by the user initially was not the image desired by the user, the present invention relates to a technique of finding and providing an image desired by the user.

In a similar image search method executed by a computer, a target image selected by a user in a clothing database in which a plurality of different clothing photographs and sketch images related to the plurality of clothing photographs are stored for each category. searching and providing a clothing picture corresponding to the image), receiving a sketch input for adding detail to the clothing picture, and sketches drawn on the provided clothing picture extracting a unified garment feature representation from a unified image comprising a sketch and the provided garment photo, and targeting the garment database, the image having the highest similarity with the unified garment feature point based on the unified garment feature point It may include the step of searching for and providing.

According to one aspect, the integrated clothing feature point, at least one feature point 1 extracted for the entire image of the clothing picture, at least one feature point 2 extracted from the partial image of the clothing picture corresponding to a predetermined specific element, sketch At least one feature point 3 extracted for the entire image of the image, and at least one feature point 4 extracted from the partial image of the sketch image corresponding to a predetermined specific element.

According to another aspect, a dip based on keyword labeling data applied to each image and similarity data evaluated by a user for each of the garment photo, the partial image of the garment photo, the sketch image, and the partial image of the sketch image The method may further include a learning step of performing deep learning.

According to another aspect, the learning step of performing the deep learning, performing a deep learning to obtain the feature point 1 and the feature point 2 for the garment photo and the partial image of the garment photo, and the feature point Separately from deep learning for acquiring 1 and feature point 2, deep learning for acquiring feature points 3 and 4 may be performed.

According to another aspect, the specific element may include different elements according to the category to which the target image belongs.

According to another aspect, the searching and providing of the image having the highest similarity with the integrated image may include searching for at least one matching image matched by keyword labeling data applied to the integrated image with respect to the clothing database. And determining an image having the highest similarity with the integrated image based on an Euclidean distance between the at least one matching image and the integrated image found.

According to another aspect, the determining of the image having the highest similarity with the unified image may include: Euclidean distance between the entire image of the matching image and the whole image of the unified image, based on the searched at least one matching image. ) Can be calculated.

According to another aspect, the determining of the image having the highest similarity with the unified image may correspond to specific keyword labeling data among a plurality of keyword labeling data applied to the unified image, based on the searched at least one matching image. An Euclidean distance between the partial image of the matching image and the partial image of the integrated image corresponding to the specific keyword labeling data may be calculated.

According to another aspect, the determining of the image having the highest similarity with the unified image may be based on a sum of Euclidean distances calculated for a plurality of keyword labeling data for a specific matching image among the at least one matching image. By setting the final similarity of the specific matching image calculated based on the input parameter of the learning model, the image having the highest similarity may be determined.

The present invention combines a sketch drawn by a user using a touch tool (for example, a touch fan, a hand, etc.) on a garment photo and a photo, based on a learning model (CNN model) trained through deep learning. By searching for and providing an image similar to the combined image of the sketch, a user may provide a desired image more quickly without checking all the plurality of images stored in the database until the user finds the correct image. That is, the search time required for the user to find the desired image can be shortened, and the accuracy of providing an image more suitable for the desired design can be increased.

In addition, by repeatedly searching for an image similar to a single image including a sketch, a sketch and a photo on the similar image provided through the search until the image desired by the user is provided (that is, performing a refinement), Even if the image initially selected by the user is not an image desired by the user, or a slightly similar image, the user may eventually find and provide the desired image.

1 is a block diagram illustrating an internal configuration of a similar image search system for performing a similar image search method according to an embodiment of the present invention.
2 is a flowchart illustrating a method of searching for a similar image according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram illustrating a screen configuration for explaining an operation of searching for and providing a similar image based on a combination of a photo and a sketch, according to an exemplary embodiment.
FIG. 4 is a diagram provided to explain an operation of performing learning based on a clothes photograph full image and a partial image according to an embodiment of the present invention.
FIG. 5 is a diagram provided to explain a method of searching for an image matching an integrated image based on integrated clothing feature points according to an embodiment of the present disclosure.
6 is a diagram illustrating an operation of calculating a similarity degree based on triplet ranking data.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention relate to a technique of searching for and providing a similar image by combining a sketch drawn on the basis of an input of a user terminal (that is, a user) on the photo, in particular, clothing photos and sketches stored in a database. The present invention relates to a technique for performing deep learning on images to search for similar images based on feature representations extracted from the learned images.

In the present embodiments, the user terminal represents an electronic device including a desktop PC or the like carried by the user, such as a smartphone, a tablet, a laptop, or the like, and includes a wired or wireless Internet, or the like. Can support

In the present embodiments, the "clothing photo" represents a photographic image of a photograph of an existing costume product. For example, the clothing image related to the clothing product that is actually sold in an online or offline store is used for deep learning. Can be collected. The collected clothing photographs are classified according to a category (ie, clothing group) to which the corresponding clothing belongs, and the identifier information of the clothing photograph is associated with at least one keyword labeling data used when collecting the corresponding clothing photograph to the clothing database. Can be stored and maintained.

1 is a block diagram illustrating an internal configuration of a similar image search system according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a similar image search method according to an embodiment of the present invention.

The similar image retrieval system 100 according to the present exemplary embodiment may include a processor 110, a bus 120, a network interface 130, a memory 140, and a clothing database 150. The memory 140 may include an operating system 141 and a service providing routine 142. The processor 110 may include a learning controller 111, a search provider 112, a user interface provider 113, and a feature point extractor 114. In other embodiments, similar image retrieval system 100 may include more components than the components of FIG. 1. However, it is not necessary to clearly show most of the prior art components. For example, similar image retrieval system 100 may include other components, such as a display or a transceiver.

The memory 140 is a computer-readable recording medium, and may include a permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. In addition, the memory 140 may store program codes for the operating system 141 and the service providing routine 142. These software components may be loaded from a computer readable recording medium separate from the memory 140 using a drive mechanism (not shown). Such a separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, and the like. In other embodiments, software components may be loaded into memory 140 via network interface 130 rather than on a computer readable recording medium.

The bus 120 may enable communication and data transmission between components of the similar image retrieval system 100. The bus 120 may be configured using a high-speed serial bus, a parallel bus, a storage area network and / or other suitable communication technology.

The network interface 130 may be a computer hardware component for connecting the similar image retrieval system 100 to a computer network. The network interface 130 may connect the similar image retrieval system 100 to a computer network through a wireless or wired connection.

The clothing database 150 may include a plurality of clothing pictures belonging to different categories, a partial image corresponding to at least one specific element corresponding to each of the clothing pictures (ie, a partial image of the clothing picture), and the plurality of clothing pictures. The sketch images corresponding to the respective images and the partial images corresponding to the specific elements corresponding to the respective sketch images (that is, the partial images of the sketch) may be classified according to categories to be stored and maintained in association with the identifier information of the corresponding clothing picture. . In this case, the specific element is pre-specified information reflecting the characteristics of the corresponding clothing, targeting the entire clothing photo desired by the user (that is, the entire image of the clothing photo desired by the user), and a part of the clothing photo. It can be used to detail search for the corresponding partial image.

Here, the specific element may include different elements for each category, or may be pre-specified to include different elements for each clothing picture belonging to the same category. For example, when the category to which the selected clothing photograph belongs belongs to jeans, the specific element may be pre-designated to include individual elements such as pockets, washes, and disengagements. In the case of a wedding dress, the particular element can be pre-specified to include a neckline, and in the case of a jacket, the pocket, neckline, etc. can be pre-specified to include individual elements.

The garment database 150 may be included in the similar image retrieval system 100 or may exist as an external database built on another system separate from the similar image retrieval system 100. Similar image search system 100 may be implemented in a platform (platform) form to search for and provide a similar image to a user terminal connected to the server, an application (ie, a service that provides a similar image search on the user terminal) App).

The processor 110 may be configured to process instructions of a computer program by performing input / output operations of the basic arithmetic, logic, and similar image retrieval system 100. The instructions may be provided to the processor 110 by the memory 140 or the network interface 130 and via the bus 120. The processor 110 may be configured to execute program code for the learning controller 111, the search provider 112, the user interface provider 113, and the feature point extractor 114. Such program code may be stored in a recording device, such as memory 140.

The learning control unit 111, the search providing unit 112, the user interface providing unit 113, and the feature point extracting unit 114 may be configured to perform the steps 210 to 250 of FIG. 2.

In operation 210, the learning control unit 111 may perform deep learning based on keyword labeling data applied to each image of the garment photo, the partial image of the garment photo, the sketch image, and the partial image of the sketch image. Can be done.

For example, the learning control unit 111 targets a plurality of clothing photographs that are collected and stored and maintained in the clothing database 150, and partial images of each of the plurality of clothing photographs, based on the entire image of the clothing photograph. May be obtained and deep learning may be performed to acquire a feature point based on the partial image of the clothing picture. In addition, the learning controller 111 targets a plurality of sketch images collected and stored and maintained in the clothing database 150, and partial images of each of the plurality of sketch images, based on the entire image of the sketch image. Acquisition may be performed, and deep learning may be performed to acquire a feature point based on the partial image of the sketch image.

That is, the learning control unit 111 may separately perform the deep learning by dividing the clothing photograph and the sketch image in order to extract optimized feature points for each of the clothing photograph and the sketch image.

In operation 220, the search provider 112 targets the garment image 150 stored by the user to the garment database 150 in which the plurality of clothes images and the sketch images associated with the clothes images are stored for each category. ) Can be provided by searching for a photo of the clothing corresponding to).

For example, as an application (ie, a service app) for providing a similar image search service is executed, a screen for selecting a category of clothing to be provided by a user may be displayed on a user terminal that is a similar image search system 100. Can be. In this case, when jeans are selected through the screen, jeans-related images corresponding to the selected category among the clothes pictures stored in the clothing database 150 may be displayed on the screen. When there is gender information (eg, women, men, etc.) based on the registered user information at the time of membership registration, women's jeans-related images may be displayed on the screen. In addition to gender information, when age information is registered, a keyword in which age information is combined in addition to a combination of a selected category and gender information is keyword labeling data. It may be searched and displayed on the screen.

As such, when a target image suitable for a design / style desired by the user is selected among the clothing photos displayed on the screen, a screen on which a plurality of clothing photos corresponding to the selected category are displayed is displayed on the full screen. The screen can be switched to display.

In operation 230, the user interface providing unit 113 may receive a sketch input for adding a detail to the clothing picture corresponding to the target image by using a tool such as a touch fan. For example, a sketch reflecting the contents drawn by the user using a tool such as a touch pan on the target image displayed on the screen may be displayed on the screen.

In operation 240, the feature point extractor 114 may extract the integrated clothing feature point from the integrated image including the sketch and the clothing picture drawn on the clothing picture corresponding to the target image. Here, the integrated clothing feature point corresponds to at least one feature point 1 extracted from the entire clothing picture corresponding to the integrated image in which the target image and the sketch are combined, and a specific element predetermined in association with the clothing picture corresponding to the target image. At least one feature point 2 extracted from the partial image (ie, the partial image of the garment photograph), at least one feature point 3 extracted from the entire sketch image corresponding to the integrated image, and a predetermined specific specification of the target image. At least one feature point 4 extracted from the partial image of the sketch image corresponding to the element may be included.

In operation 250, the search provider 112 may search for and provide an image having the highest similarity to an integrated image based on the extracted integrated clothing feature point, based on the learned clothing photos stored in the clothing database 150. .

FIG. 3 is a diagram illustrating a screen configuration for explaining an operation of searching for and providing a similar image based on a combination of a photo and a sketch, according to an exemplary embodiment.

Referring to 310 of FIG. 3, when a wedding dress is selected from the categories, a plurality of clothing pictures corresponding to the wedding dress of the category may be retrieved from the clothing database 150 and displayed on the screen. If a user selects a photo of a plurality of wedding dresses-related clothing images similar to the ones he / she wants by touching the screen or selecting a check box, the clothing photo corresponding to the selected wedding dress (that is, the wedding dress photo) is displayed. May be displayed.

Referring to the screen 320, a design desired by the user may be sketched using a touch fan or the like on the wedding dress photo displayed on the screen 310. The sketch drawn on the wedding dress photo may be preset in red, blue, and the like to distinguish the wedding dress, and the preset basic colors, such as red and blue, may be changed by user selection. For example, although red is previously designated as a basic color, it may be changed to green or purple by user selection.

As such, when a sketch is added with the desired detail on the wedding dress picture, the search provider 112 matches an image (that is, the wedding dress picture and the sketch combined image in the clothing database 150). , Matching images). In this case, the search provider 112 may search the matching image for the image having the highest similarity with the integrated image among the clothing photos stored in the clothing database 150 based on the integrated clothing feature points extracted from the integrated image.

For example, the search provider 112 may search the clothing database 150 for at least one matching image matched by keyword labeling data provided to the integrated image. For example, the keyword labeling data may include a category, gender information, and a predetermined specific element (eg, neckline, pocket, washing, etc.) reflecting the characteristics of the corresponding garment. Here, the specific element may be set differently according to the category to which the corresponding clothing belongs, and in the case of the wedding dress, the specific element may include a neckline. Accordingly, the keyword labeling data given to the wedding dress may include a woman, a wedding dress, a neckline, and the like. Then, the search provider 112 searches for at least one matching image that matches the unified image based on the keyword labeling data assigned to each of the clothing photos stored in the clothing database 150 and the keyword labeling data assigned to the wedding dress. can do. The search provider 112 may calculate an Euclidean distance between each matching image and the integrated image based on the integrated clothing feature points extracted from each of the searched matching images and the integrated clothing feature points of the integrated image. have. In addition, the search provider 112 may set each of the calculated Euclidean distances as input values of the learning model to derive a similarity value between each of the matching images and the integrated image through the learning model. In this case, as the similarity value among the matching images is relatively large, it may correspond to an image similar to the integrated image. The search provider 112 may provide the matching images on the screen by arranging the similarity values in ascending order. For example, as shown in screen 330, a matching image having the highest similarity with the integrated image may be displayed. As such, any one image having the highest similarity may be displayed on the screen, or a plurality of matching images arranged in the order of high similarity may be displayed on the screen. When a plurality of matching images are provided, if any one image is selected, only the selected matching image may be displayed on the entire screen.

In this case, if the image displayed on the screen 330 is not a wedding dress of a desired design, the user interface providing unit 113 may additionally receive a sketch input on the wedding dress photo displayed on the screen 330. Then, the search providing unit 112 generates an integrated image combining the additionally input sketch and the wedding dress picture of the screen 330 as shown in the screen 340, and at least one matching the integrated image based on keyword labeling data of the integrated image. Search for a matching image. The integrated sketches of the screen 340 are combined based on the Euclidean distance calculated based on the unified clothing feature points extracted from the unified image and the unified garment feature points extracted from each of the searched matching images. The matching image having the highest similarity with the image may be determined. The matching image with the highest similarity may be displayed on the screen 350. In this case, if the wedding dress picture displayed on the screen 350 corresponds to the design desired by the user, the adding and searching of the sketch is terminated. If the wedding dress picture does not correspond to the desired design, the sketch is added to the wedding dress photo displayed on the screen 350 The refine process of searching for matching wedding dress photos based on the integrated image combining the sketches and the photographs may be repeated. That is, the search provider 112 adds a sketch to an image output as a search result until a design desired by a user is provided based on an interactive search method (that is, until additional sketch input is not received). And refine the search.

FIG. 4 is a diagram provided to explain an operation of performing learning based on a clothes photograph full image and a partial image according to an embodiment of the present invention.

Referring to 410 of FIG. 4, an operation of performing deep learning based on the entire garment photo image will be described. Referring to 450, the partial image corresponding to each specific element predetermined in the garment photo will be described. An operation of performing deep learning will be described.

The learning control unit 111 may perform deep learning on a plurality of clothes pictures stored in the clothes database 150. Here, the identifier information of each of the plurality of garment photos may be stored and maintained in the garment database 150 in association with keyword labeling data assigned to each garment photo.

Referring to 410, the learning controller 111 may perform deep learning on an entire image of a specific clothing picture (eg, jeans) to be learned. In this case, the learning control unit 111 may learn a specific garment photo based on keyword labeling data such as a category, a specific element, etc. to which the specific garment photo belongs. For example, in the case of jeans, the keyword labeling data may include women's clothing, jeans, pockets, washes, and tears. Depending on the jeans, the washing level is different, and there may be jeans with and without jeans. Accordingly, the learning controller 111 may perform deep learning based on specific factors such as pockets, washes, and tears in addition to categories such as women's clothing and jeans. Based on the CNN (Convolution Neural Network) model generated based on the training data (eg, keyword labeling data), feature representations may be extracted for a specific garment photo (the entire image of the jeans photo). . In this case, the learning control unit 111 may calculate a probability value with respect to all the specific elements (eg, pockets, washes, and dissolutions) corresponding to the jeans picture, based on the extracted feature points. When the calculated probability value is greater than or equal to a predetermined reference value, the specific element exists in the corresponding jeans picture, and when the calculated probability value is smaller than the reference value, it may be determined that the specific element does not exist in the jeans picture. For example, when a probability value related to a pocket is greater than or equal to a reference value, a specific jeans photograph may be determined to have a pocket present, and when smaller than the reference value, it may be determined to be a jeans that do not include a pocket. In addition, if the washing-related probability value is greater than or equal to the reference value, the washing may be determined to exist, and if it is smaller than the reference value, the jeans may not be included. As such, the presence or absence of a specific element related to a specific garment photo may be determined according to the magnitude of the probability value.

As such, the learning control unit 111 may perform deep learning on an entire image of a specific clothing picture (eg, a jeans picture) and then perform deep learning on a partial image for optimization.

Referring to 420, the learning controller 111 may perform deep learning based on the feature points extracted from the partial image 421 corresponding to the pocket in the jeans picture rather than the entire image of the jeans picture such as 410. That is, the learning controller 111 may learn the visual characteristics of the pocket using only the partial image corresponding to the pocket. Similarly, only the partial image 422 corresponding to the solution can be used to learn the visual characteristics of the solution.

Similar to the garment photo, the learning controller 111 may perform deep learning on sketch images stored and maintained in the garment database 150. In this case, referring to 430, the learning control unit 111 may perform deep learning based on the entire sketch image 431 corresponding to a specific clothing picture (eg, a jeans picture). In addition, the learning controller 111 may perform deep learning based on a partial image (eg, a partial image of a sketch corresponding to termination) 432 corresponding to each specific element that is predetermined in a specific garment photo (eg, a jeans photo). Can be done.

FIG. 5 is a view provided to explain a method of searching for an image matching an integrated image based on integrated clothing feature points, and FIG. 6 is based on triplet ranking data. Is a diagram illustrating an operation of calculating similarity.

Referring to 510 of FIG. 5, the search provider 112 may calculate keyword labeling data (eg, women's clothing, jeans, pockets, washes, and tears) corresponding to a specific garment image 511 and specific calculations calculated for each keyword labeling data. The similarity between the two garment images can be calculated based on the probability value 512 between the garment image 511 and the image that is supposed to match the specific garment image 511 (eg, the garment image with the same keyword labeling data specified). . Here, the probability value 512 may represent a matching probability between two images. For example, the search provider 112 may calculate a Euclidean distance between the specific garment image 511 selected by the user and the matching image based on Equation 1 below.

[Equation 1]

According to Equation 1 above, the search provider 112 is based on the integrated clothing feature points extracted for the entire image of the specific clothing image 511 and the integrated clothing feature points extracted for the entire image of the matching image. The calculated Euclidean distance can be calculated. Here, a plurality of matching images may exist, and the search provider 112 may calculate Euclidean distance to determine similarity between the plurality of matching images and the image 511.

In this case, referring to 520, the search provider 112 may analyze the similarity of a partial image corresponding to a specific element among keyword labeling data of a specific clothing image (ie, a combined image of a combination of a photo and a sketch) for optimization. Can be calculated.

For example, the search provider 112 may be configured based on the feature points extracted from the partial image corresponding to the pocket of the image 511 and the feature points extracted from the partial image corresponding to the pocket of each of the plurality of matching images. The Euclidean distance between the partial image corresponding to the pocket of the image 511 and the partial image corresponding to the pocket of the plurality of matching images may be calculated. For example, the Euclidean distance (eg, 0.61) between the partial image corresponding to the pocket of the garment image 511 and the partial image 521 of the matching image 1, and the Euclidean distance (eg, 0.84) with the partial image 522 of the matching image 2 , Euclidean distance with matching image 3 523 (eg, 0.57), and the like may be calculated. Euclid distance with each of the matching images may be calculated for each specific element corresponding to the integrated image with respect to each of the other specific elements other than the pocket, washing and dissolution. That is, the search provider 112 may calculate the Euclidean distance indicating the similarity between the integrated image and the matching image for each specific element based on Equation 2 below.

[Equation 2]

The most similarity to the unified image based on the Euclidean distance calculated for the entire image of the unified image based on Equation 1 above and the Euclidean distance calculated for the partial image corresponding to a specific element of the unified image based on Equation 2 The high matching image can be determined. In this case, the matching image with the highest similarity may be provided through the screen, or after the matching images are arranged in the order of high similarity, the plurality of aligned matching images may be displayed on the screen together with the corresponding similarity.

For example, when the Euclidean distance is calculated for each partial image corresponding to a specific element, the final image between the matching image (eg, Matching image 1) and the integrated image based on the Euclidean distance calculated for each partial image as shown in Equation 3 below. Similarity D _total can be calculated.

[Equation 3]

According to Equation 3, before the final similarity of a specific matching image is set as an input parameter of the learning model, the final similarity is the sum of Euclidean distances D _all , D _pocket , D _wash , and D _tear of the matching image. It can be calculated based on.

In this case, the similarity according to the sum of Euclidean distance calculated based on Equation 3 above, and the similarity between the integrated image and the matching image according to the evaluation of real people may be different, and as the difference between the similarities increases, the loss is increased. For example, referring to 610 of FIG. 6, the search provider 112 determines that Euclidean distance between the integrated image 611 and the matching image 1 612 is matched with the matching image 2 613. If greater than Euclid distance, it may be determined that matching image 1 612 is similar to unified image 611 than matching image 2 613, but in practice, people are more similar to unified image 611 and matching image 2 613. I can feel it.

Accordingly, the learning controller 111 may reflect the similarity between the integrated image 611 and the matching images based on triplet ranking data collected from other user terminals to reflect the user's response. Can be. For example, the learning control unit 111 may train the learning model in a direction in which a difference between the similarity evaluation of real people and the similarity evaluation mechanically calculated in the system based on Equation 3, that is, loss is reduced. have. In this case, the learning controller 111 may set the final similarity for each matching image calculated based on Equation 3 as an input parameter of the learning model to train the learning model (ie, the CNN model) in a direction of reducing loss. Can be.

As such, the similarity derived by learning through the learning model may be expressed by Equation 4 below.

[Equation 4]

The similarity formula represented by Equation 4 may be updated and improved whenever training data (ie, triplet ranking data) is increased or changed.

In this way, the user response to the search results provided by the system can be reflected in the learning model (CNN model) to reduce the loss between the matching image that users perceive as more similar to the integrated image and the matching image calculated by the system. have. For example, the loss may be calculated based on Equation 5 below. Here, the triplet ranking data is response data collected from users (ie, user terminals), and present an integrated image and a plurality of matching images, and the image most similar to the integrated image when viewed by the human eye among the presented images. May include identifier information of the selected matching image.

[Equation 5]

In Equation 5 above, anchor is an integrated image 621, positive is a matching image 623 among the plurality of matching images that users perceive as most similar to the integrated image 621, and negative is an integrated image in the system. The matching image 622 determined to have the highest similarity with 621 may be represented. That is, negative may correspond to a matching image having the largest Euclidean distance with the integrated image 621 based on Equations 3 and 4 above.

According to Equation 5, the unified clothing feature points of the unified image 621 and the unified garment feature points and the matching image 622 of the unified clothing feature points of the unified image 621 and the unified garment feature points of the unknown image 623 are calculated. Loss may be calculated based on the Euclidean distance calculated based on the unified garment feature points of h). Then, the learning controller 111 may perform learning based on the CNN model to minimize the calculated loss. As such, feature values of images matching the integrated image provided as a search result are applied to the input of the CNN model for learning, and as the learning proceeds in a direction that minimizes the loss, The CNN model can be trained in such a way that Euclid distances calculated based on Equation 3 better reflect the similarity perceived by users than before considering loss. As the learning is repeated, the integrated image 611 and the matching image 1 612 initially provided the search results with the highest similarity. However, as the learning considering the loss is performed, the integrated image 621 and the matching image 2 623 are performed. ) May provide the search results with the highest similarity.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (6)

In a similar image retrieval method executed by a computer,
A clothing image corresponding to a target image selected by a user is searched and provided in a clothing database in which a plurality of different clothing pictures and sketch images related to the plurality of clothing pictures are stored for each category. step;
Receiving a sketch input for adding detail on the garment photo, targeting the provided garment photo;
Extracting an integrated garment feature representation from an integrated image comprising a sketch drawn on the provided garment photo and the provided garment photo; And
Searching for and providing an image having the highest similarity with the integrated image based on the integrated clothing feature point in the clothing database;
Similar image search method comprising a. The method of claim 1,
The integrated clothing feature point may include at least one feature point 1 extracted from an entire image of a garment photo, at least one feature point 2 extracted from a partial image of the garment photo corresponding to a predetermined specific element, and an entire image of a sketch image. At least one feature point 3 extracted as a target, and at least one feature point 4 extracted from a partial image of the sketch image corresponding to a predetermined specific element,
The specific element may include different elements according to a category to which the target image belongs.
Similar image search method characterized in that. The method of claim 2,
A learning step of performing deep learning based on keyword labeling data applied to each image of the garment photo, the partial image of the garment photo, the sketch image, and the partial image of the sketch image.
More,
The learning step of performing the deep learning,
Performing deep learning to acquire the feature points 1 and 2 from the garment photo and the partial image of the garment photo; And
Performing deep learning to obtain the feature points 3 and 4 separately from the deep learning to obtain the feature points 1 and 2;
Similar image search method comprising a. The method of claim 1,
Searching for and providing an image having the highest similarity with the integrated image,
Searching for the at least one matching image to which the keyword labeling data assigned to the integrated image matches the clothing database; And
Determining an image having the highest similarity with the integrated image based on an Euclidean distance between the retrieved at least one matching image and the integrated image
Including,
Determining the image having the highest similarity with the integrated image,
Calculating an Euclidean distance between the entire image of the matching image and the entire image of the unified image, based on the retrieved at least one matching image;
Similar image search method characterized in that. The method of claim 4, wherein
Determining the image having the highest similarity with the integrated image,
The partial image of the matching image corresponding to the specific keyword labeling data among the plurality of keyword labeling data applied to the integrated image and the partial image of the integrated image corresponding to the specific keyword labeling data are searched for the at least one matching image searched. To calculate Euclidean distance
Similar image search method characterized in that. The method of claim 4, wherein
Determining the image having the highest similarity with the integrated image,
The final similarity of the specific matching image calculated based on the sum of Euclidean distances calculated for each of the plurality of keyword labeling data is set as an input parameter of the training model, based on the specific matching image among the at least one matching image. Determining the image with the highest similarity
Similar image search method characterized in that.

Images