JP2022044687A - Image search method and apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image search apparatus and a method thereof.

SOLUTION: An image search method includes: a step of obtaining reference feature vectors corresponding to multiple reference images containing the same attribute; a step of obtaining a representative feature vector corresponding to the same attribute from a generative model trained based on the obtained reference feature vectors; and a step of comparing the feature vector of a target image stored in a database with the representative feature vector, so as to assign attribute tags corresponding to the same attribute to at least some of the target images.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to the technical field of search, and more specifically to an image-based search device and a method thereof.

With the development of information and communication technology, information retrieval using various terminals such as smartphones and personal computers is widely used. In the past, documents and the like were searched and provided based on the text entered by the user, but recently, the number of cases of searching for images such as photographs and pictures is increasing.

Searching for images such as clothing and landscapes by text alone is not easy. For example, when a user searches for clothing in the style he / she desires, it may be difficult to find a suitable image by text search.

Recently, a service has been tried to search for images similar to the search image by inputting a search image that is not text from the user, but for that purpose, for many images stored in the database, one by one. It comes with the difficulty of having to manually assign tags.

Korean Registered Patent No. 10-178521

An image search device according to an embodiment of the present invention and a method thereof have a technical problem of making it possible to easily assign attribute tags to images stored in a database.

Further, the image search device according to one embodiment and the method thereof have a technical problem of being able to more accurately search for an image of a type desired by a user by using a representative feature vector.

The image search method according to one embodiment corresponds to the same attribute from the stage of acquiring the reference feature vector corresponding to a plurality of reference images including the same attribute and the generation model trained based on the acquired reference feature vector. At the stage of acquiring the representative feature vector to be used, the feature vector of the target image stored in the database is compared with the representative feature vector, and at least a part of the target images corresponds to the same attribute. It may include the stage of assigning attribute tags.

In the image search method according to one embodiment, a stage of receiving a plurality of search images from a user terminal, a stage of acquiring a feature vector corresponding to the plurality of search images, and a step of training a generative model with the acquired feature vectors. The stage, the representative feature vector output from the generated model trained by the feature vector, and the feature vector of the target image stored in the database are compared, and the target image corresponding to the plurality of search images is determined. It may include a step and a step of transmitting the determined target image to the user terminal.

An image retrieval device according to an embodiment includes a processor, at least one program, a memory for storing a generation model, and a database for storing a target image, wherein the processor is executed with the at least one program. By doing so, a reference feature vector corresponding to a plurality of reference images including the same attribute is acquired, and a representative feature vector corresponding to the same attribute is acquired from the generated model trained based on the acquired reference feature vector. Then, the feature vector of the target image and the representative feature vector can be compared, and an attribute tag corresponding to the same attribute can be assigned to at least a part of the target images of the target images.

In the image search method according to another embodiment, a step of connecting to an image search device and selecting an attribute and an attribute tag corresponding to the selected attribute among the target images stored in the image search device are assigned. The target image stored in the image search device includes a step of receiving the target image from the image search device, and the target image stored in the image search device includes a representative feature vector corresponding to each of a plurality of attributes and a feature vector of the target image. Based on the comparison result, at least a part of the attribute tags corresponding to the plurality of attributes is assigned.

The image search device according to the embodiment of the present invention and the method thereof can easily assign attribute tags to images stored in a database.

Further, the image search device and the method thereof according to the embodiment of the present invention can use the representative feature vector to search the image of the kind desired by the user more accurately.

However, the effects that can be achieved by the image search device and the method thereof according to one embodiment of the present invention are not limited to those described above, and other effects not mentioned are described in the following description. It will be clearly understood by those skilled in the art in the technical field to which the disclosure belongs.

It is a drawing which illustrates the image search apparatus and the user terminal by one Embodiment. It is a flowchart for demonstrating the method of allocating the attribute tag by one Embodiment. It is a drawing for demonstrating the method of acquiring the representative feature vector from a generative model. It is an exemplary drawing for demonstrating the method of assigning an attribute tag to a target image. It is a drawing which illustrates the UI (user interface) screen which allows a user to select an attribute. It is a flowchart for demonstrating the image search method by one Embodiment. It is a flowchart for demonstrating the image search method by another embodiment. It is a drawing which shows the feature vector by attribute extracted from the search image. It is a drawing for demonstrating the method of exchanging at least one of the attribute-specific feature vectors of a search image with a representative feature vector. It is a drawing for demonstrating the method of comparing the attribute-specific feature vector of a search image, and the attribute-specific feature vector of a target image. It is an exemplary drawing which illustrates the search image which contains a plurality of objects. 11 is a drawing for explaining a method of exchanging at least one of the feature vector sequences of the search image shown in FIG. 11 for a representative feature vector. FIG. 11 is an exemplary drawing illustrating a search result image corresponding to the search image illustrated in FIG. It is a block diagram which illustrates the structure of the image search apparatus by one Embodiment.

Although various modifications can be made and various embodiments can be included in the present disclosure, specific embodiments are illustrated in the drawings, which will be described in detail. However, it should be understood that the present disclosure is not limited to a particular embodiment and includes all modifications, equivalents or alternatives contained within the ideas and technical scope of the present disclosure.

In the description of the present embodiment, if it is determined that the specific description of the related publicly known technology makes the gist unnecessarily unclear, the detailed description thereof will be omitted. Further, the number or ordinal number (for example, first, second, etc.) used in the description process of the present embodiment is merely an identification symbol for distinguishing one component from another component.

Further, in the present specification, when one component is referred to as being "connected" or "connected" to another component, the one component is referred to as the other component. It must be understood that they are directly connected or even directly connected, but are connected or even connected via other components in the middle, as long as there is no particular opposite statement. It is.

Further, in the present specification, the components expressed as "-part (unit)" and "module" are such that two or more components are combined into one component, or one component is further subdivided. It is also differentiated into two or more according to the converted functions. In addition to the stock functions that each component is in charge of, each of the components described below can also perform some or all of the functions that other components are in charge of, and each component is in charge of the main. It goes without saying that some of the functions are performed even if they are exclusively occupied by other components.

Hereinafter, embodiments based on the technical ideas of the present disclosure will be described in detail in order.

FIG. 1 is a drawing illustrating an image search device 100 and a user terminal 10 according to an embodiment.

The user terminal 10 selects or inputs a search condition (for example, a search image, an attribute, etc.) to the image search device 100 and transmits the image, and the image search device 100 transmits a search result image corresponding to the search condition to the user terminal 10. To transmit to. In one embodiment, the image search device 100 can store a target image for determining a search result image corresponding to the search condition in the database. The image search device 100 can transmit the target image corresponding to the search condition among the target images to the user terminal 10 as a search result image.

The user terminal 10 may include, but is not limited to, a desktop PC (personal computer), a tablet, a smartphone, a notebook computer, and the like that can be connected to the image search device 100 via a network.

In one embodiment, the image search device 100 and the user terminal 10 are connected to each other via a network and can transmit and receive data to and from each other. Here, the network may include a wired network and a wireless network, and specifically, a short-range network (LAN: local area network), a metropolitan area network (MAN: metropolitan area network), and a wide area network (WAN:). It may include various networks such as wide area network). Further, the network 10 may include a known worldwide web (WWW: worldwide web). However, the network 10 according to the present invention is not limited to the networks listed above, and may include a known wireless data network, a known telephone network, and a known known wireless television network as at least a part thereof.

The image search device 100 according to one embodiment can assign an attribute tag to a target image stored in advance. As used herein, the term "attribute" means the characteristics of the objects contained in the image. The attributes are classified by category. For example, a piece of clothing can have a red attribute that belongs to the hue category, a striped pattern attribute that belongs to the pattern category, and a one-piece attribute that belongs to the shape category. Further, in the present specification, the “attribute tag” means information assigned to an image to indicate the characteristics of an object in the image. For example, an attribute tag of a piece of clothing can indicate that the hue of the piece of clothing is red, the pattern is striped, and the shape is one piece.

A method of assigning an attribute tag to a target image by the image search device 100 will be described with reference to FIG.

FIG. 2 is a flowchart for explaining a method of assigning attribute tags according to an embodiment.

At the stage S210, the image search device 100 acquires reference feature vectors corresponding to a plurality of reference images having the same attribute or to which the same attribute tag is assigned. In one embodiment, the administrator can assign the same attribute tag to a plurality of reference images. For example, multiple reference images are assigned the same attribute tag, such as "red," "skewers," or "clothes popular with women."

The reference feature vector corresponding to the plurality of reference images may be output by the machine learning model. The machine learning model may include, for example, a CNN (convolutional neural network). Further, the reference feature vector may be N (N is an integer of 1 or more) dimension data. In one embodiment, the reference feature vector corresponding to the plurality of reference images is also generated by post processing after being output by the machine learning model. In the post-processing, for example, after arranging a plurality of elements contained in the feature vector output from the machine learning model on a size basis, an index indicating the position of the upper k elements in the feature vector is acquired and acquired. It may include a process of generating an index feature vector having the index as an element as a reference feature vector.

In the S220 step, the image search device 100 acquires a representative feature vector corresponding to the same attribute or the same attribute tag from the generated model trained based on the reference feature vectors corresponding to the plurality of reference images. The generative model can generate a representative feature vector similar to the reference feature vector by training based on the reference feature vectors of the plurality of reference images. In one embodiment, the generative model may include a generative adversarial network (GAN).

Referring to FIG. 3, the generative model 300 is trained by the reference feature vectors 320a, 320b, 320c corresponding to the reference images 310a, 310b, 310c, and then, for example, input random noise as input data and the reference feature vector. Representative feature vectors similar to 320a, 320b, 320c can be generated. In one embodiment, the GAN corresponding to the generative model 300 is composed of a plurality of FCLs (fully connected layers). FCL is a network structure in which each layer is composed of at least one node, but each node of one layer is individually connected to all the nodes contained in the previous layer and the next layer. .. Since the GAN is trained by a reference feature vector that is not an image, it is not necessary to include a CNN, and the FCL alone can generate a representative feature vector similar to the reference feature vector. That is, by using a GAN composed of a plurality of FCLs for generating a representative feature vector instead of a GAN for image generation, training is speeded up and the processing speed is also very fast.

In one embodiment, the generative model can also acquire an average value of a reference feature vector element and generate a representative feature vector having the acquired average value as an element.

Referring to FIG. 2 again, in the step S230, the image search device 100 compares the feature vector of the target image stored in the database with the representative feature vector. In one example, the image search device 100 can calculate the distance (for example, cosine distance) between each feature vector of the target image and the representative feature vector. Since the distance calculated for determining the similarity between the feature vectors is known, detailed description thereof is omitted in the present specification.

At the stage S240, the image search device 100 assigns an attribute tag corresponding to the same attribute possessed by the plurality of reference images to at least a part of the target images among the target images. In one embodiment, the image search device 100 can assign an attribute tag corresponding to the same attribute to a target image corresponding to a feature vector having a distance equal to or less than a preset value in comparison with a representative feature vector.

FIG. 4 is a drawing for explaining a method of assigning an attribute tag to a target image. With reference to FIG. 4, the feature vectors 410b, 420b, 430b of the target images 410a, 420a, 430a and the representative feature vector are shown. According to the comparison result, the attribute tags 410c and 420c of the "zigzag pattern" are assigned to the first target image 410a and the second target image 420a, and the attribute tag 430c of the "pergandy hue" is assigned to the third target image 430a. Although FIG. 4 is shown so that one attribute tag is assigned to one target image, a plurality of attribute tags may be assigned to one target image.

According to one embodiment, attribute tags can be assigned to a large number of target images based on a relatively small number of reference images, so that the administrator can assign attribute tags to each target image stored in the database. Eliminates the need to assign attribute tags.

Further, according to one embodiment, even when it becomes necessary to assign a new attribute tag to the target image due to the development of a new attribute, fashion, etc., the new attribute tag can be quickly assigned to the target image. can. For example, if the administrator selects a small number of reference images with new attributes, the image search device 100 trains the generative model with the reference feature vector of the selected reference image, and the representative output from the generative model. Based on the comparison result between the feature vector and the feature vector of the target image, the attribute tag corresponding to the new attribute can be assigned to the target image.

FIG. 5 is a drawing illustrating a user interface (UI) screen 500 that allows the user to select attributes.

As described above, if the attribute tag is assigned to the target image stored in the database, the user can easily select the specific attribute on the UI screen 500 displayed on the display of the user terminal 10. Search is possible.

Referring to FIG. 5, if the search screen 500 is displayed on the display of the user terminal 10, the user can use a part of the many attributes 530 belonging to the specific category (for example, pattern) 520 on the search screen 500 (for example, a pattern). For example, by selecting (check), the target image to which the attribute tag corresponding to the attribute is assigned can be received in the search result.

The image search device 100 can transmit to the user terminal 10 as a search result image a target image to which a tag corresponding to the attribute selected by the user terminal 10 is assigned among the target images stored in the database. ..

As an embodiment, the user can also input the search term 510 as an attribute on the search screen 500. Among the target images stored in the database, the image search device 100 can transmit the target image having the attribute tag related to the attribute corresponding to the search term 510 to the user terminal 10 as the search result image.

In one embodiment, the image search device 100 compares the representative feature vector corresponding to the attribute selected by the user terminal 10 with the feature vector of the target image stored in the database, and is it the same as the representative feature vector? , Or a target image corresponding to a similar feature vector can be transmitted to the user terminal 10 as a search result image. The image search device 100 can transmit a target image corresponding to a feature vector having a distance equal to or less than a preset value with the representative feature vector to the user terminal 10 as a search result image.

The feature vector to be compared with the representative feature vector can differ depending on what kind of machine learning model outputs the reference feature vector used to train the generative model that outputs the representative feature vector. .. For example, if the reference feature vector used to train the generative model is output by the dependent model, then the feature vector of the target image to be compared is also output by the dependent model. The dependent model means a machine learning model trained to output feature vectors corresponding to a particular category of attributes. For example, the dependent model may include a model dedicated to the hue category, a model dedicated to the pattern category, and the like. When assigning a hue attribute tag to a target image, the administrator inputs the reference image into the model dedicated to the hue category, acquires the reference feature vector corresponding to the reference image, and generates it with the acquired reference feature vector. You can train the model.

Further, when the feature vector used for training the generative model is output by the independent model, the feature vector of the target image to be compared is also output by the independent model. The independent model means a machine learning model trained to output feature vectors independent of attribute categories.

FIG. 6 is a flowchart for explaining an image search method according to an embodiment.

At the stage S610, the image search device 100 receives a plurality of search images from the user terminal 10. The image search device 100 can also directly receive a plurality of search images from the user terminal 10 via the network, and information capable of identifying the plurality of search images from the user terminal 10 (for example, a plurality of searches). It is also possible to receive a URL in which an image is stored, identification information of a plurality of search images, etc., and acquire a plurality of search images based on the received information.

At the stage S620, the image search device 100 acquires feature vectors corresponding to a plurality of search images. The image search device 100 can input a plurality of search images to the machine learning model and acquire the feature vector output from the machine learning model. The machine learning model is also a dependent model or an independent model.

At the S630 stage, the image search device 100 trains the generative model with feature vectors corresponding to a plurality of search images. When the generative model is GAN, the GAN is trained to generate data similar to the training data based on the interaction between the generator and the discriminator.

In the S640 step, the image search device 100 compares the representative feature vector output from the generated model trained with the feature vectors of the plurality of search images with the feature vector of the target image. The image search device 100 can calculate the distance between each feature vector of the target image and the representative feature vector.

As mentioned above, the feature vector to be compared with the representative feature vector is what kind of machine learning model output the feature vector used to train the generative model that outputs the representative feature vector. It also depends on.

At the S650 stage, the image search device 100 transmits the target images corresponding to the plurality of search images among the target images stored in the database to the user terminal 10 as search result images.

The image search device 100 can transmit a target image corresponding to a feature vector having a distance equal to or less than a preset value with the representative feature vector to the user terminal 10 as a search result image.

According to the embodiment illustrated in FIG. 6, for example, if the user selects a plurality of search images that match his / her taste, the image search device 100 uses the image search device 100 other than the search images that match the user's taste. The image can be provided as a search result.

FIG. 7 is a flowchart for explaining an image search method according to another embodiment.

At the S710 stage, the image search device 100 receives the search image from the user terminal 10. The image search device 100 can also directly receive the search image from the user terminal 10 via the network, and the information that can identify the search image from the user terminal 10 (for example, the URL in which the search image is stored) is stored. , The identification information of the search image, etc.) is received, and the search image can be acquired based on the received information.

At the stage S720, the image search device 100 acquires the attribute-specific feature vector of the search image. The image search device 100 can input a search image to each of a plurality of dependent models and acquire feature vectors output from the plurality of dependent models. For example, as shown in FIG. 8, the search image 810 is input to the first dependent model 821, the feature vector 831 corresponding to the first attribute (for example, the hue attribute) is acquired, and the search image 810 is obtained. The second dependent model 823 is input to acquire the feature vector 833 corresponding to the second attribute (for example, the pattern attribute), the search image 810 is input to the third dependent model 825, and the third attribute (for example, the shape attribute) is input. ) Corresponding to the feature vector 835 can be obtained.

In one embodiment, when the number of search images received from the user terminal 10 is two, any one of the search images is input to the first dependent model 821, the feature vector corresponding to the first attribute is acquired, and the other. It is also possible to input one search image of the above into the second dependent model 823 and the third dependent model 825, and acquire the feature vector corresponding to the second attribute and the feature vector corresponding to the third attribute.

In another embodiment, when the number of search images received from the user terminal 10 is three, any one of the search images is input to the first dependent model 821, and the feature vector corresponding to the first attribute is acquired. The other one search image is input to the second dependent model 823 to acquire the feature vector corresponding to the second attribute, and the other search image is input to the third dependent model 825 to correspond to the third attribute. It is also possible to acquire the feature vector to be used.

That is, the image search device 100 can combine feature vectors corresponding to a plurality of search images to acquire attribute-specific feature vectors.

FIG. 8 is illustrated such that three dependent models 821,823,825 are used to extract attribute-specific feature vectors, but this is only an example and as a search image. , Various numbers of dependent models can be used to extract feature vectors of various numbers of attributes.

Referring to FIG. 7 again, in the S730 stage, the image search device 100 uses the feature vector of the attribute selected by the user terminal 10 among the attribute-specific feature vectors as a representative corresponding to the attribute selected by the user terminal 10. Change to a feature vector. The user can select the attribute to be changed to the representative feature vector via the user terminal 10. The user terminal 10 can select an attribute to be changed (for example, a hue attribute) and another attribute to be replaced with the attribute to be changed among the attributes of the objects included in the search image. Correspondingly, the image search device 100 can change the feature vector of the attribute selected by the user terminal 10 among the feature vectors for each attribute to the representative feature vector corresponding to the other attribute.

For example, as shown in FIG. 9, in the feature vector 831 of the first attribute, the feature vector 833 of the second attribute, and the feature vector 835 of the third attribute, the feature of the second attribute selected by the user terminal 10 The vector 833 can be exchanged for the representative feature vector 933. Here, the representative feature vector 933 is also a feature vector output from the generative model trained with the reference feature vector output from the dependent model 823 dedicated to the second attribute.

At the stage S740, the image search device 100 compares the attribute-specific feature vector including the representative feature vector with the attribute-specific feature vector of the target image stored in the database. Here, the attribute-specific feature vector of the target image stored in the database is also output from the dependent model.

In one embodiment, as shown in FIG. 10, the image search device 100 includes an attribute-specific feature vector 831, 933, 835 including a representative feature vector 933, and an attribute-specific feature vector of a target image stored in a database. In comparison with 1031, 1033, 1035, feature vectors of attributes belonging to the same category can be compared with each other to obtain a distance for each attribute. The image search device 100 calculates the distance corresponding to the first attribute, the distance corresponding to the second attribute, and the distance corresponding to the third attribute for each of the attribute-specific feature vectors 1031, 1033, 1035 of the target image. Can be done.

At the stage S750, the image search device 100 can determine the target image to be transmitted to the user terminal 10 among the target images stored in the database based on the calculated distance for each attribute.

The image search device 100 determines a target image that is the same as or similar to the attribute-specific feature vector including the representative feature vector among the target images stored in the database, based on the result of combining the attribute-specific distances. can do. In one example, the image search device 100 can select a predetermined number of target images in descending order of the result value of combining the distances by attribute, or when the result value of combining the distances by attribute is equal to or more than the preset value. You can also select a target image.

In the S760 stage, the image search device 100 transmits the target image selected in the S750 stage to the user terminal 10.

According to the embodiment illustrated in FIG. 7, the user transmits the search image to the image search device 100 and changes the specific attribute of the object included in the search image to another attribute. You can receive the search results corresponding to. For example, the user may transmit a search image containing red and striped dresses to the image search device 100, select blue instead of red, and receive an image containing blue and striped dresses as a search result. You can.

FIG. 11 is an exemplary drawing illustrating a search image 1100 that includes a plurality of objects 1110, 1130, 1150.

In one embodiment, the search image 1100 transmitted from the user terminal 10 to the image search device 100 may include a plurality of objects 1110, 1130, 1150 as shown in FIG. In FIG. 11, the plurality of objects 1110, 1130, 1150 include a coat object 1110, a bottoms object 1130, and a bag object 1150. In this case, the image search device 100 crops the partial images corresponding to each of the plurality of objects 1110, 1130, and 1150 and causes them to be input to the dependent model, and the attribute-specific feature vector of each object output from the dependent model. Can be acquired. For example, the image search device 100 acquires the attribute-specific feature vector of the coat object 1110, the attribute-specific feature vector of the undergarment object 1130, and the attribute-specific feature vector of the bag object 1150 from the search image 1100 shown in FIG. can do.

The image search device 100 sets the feature vector of the specific attribute of the object selected by the user terminal 10 in the feature vector sequence including the attribute-specific feature vectors of each of the plurality of objects 1110, 1130, 1150 included in the search image 1100. It can be changed to the representative feature vector corresponding to the attribute selected by the user terminal 10. For example, when the user selects the bag object 1150 in the search image 1100 illustrated in FIG. 11 and changes the hue attribute of the bag to another hue, the image search device 100 is characterized by the hue attribute of the bag object 1150. Instead of a vector, a representative feature vector corresponding to the hue changed by the user can be included in the feature vector sequence.

For example, as shown in FIG. 12, of the attribute-specific feature vector 1210 of the first object, the attribute-specific feature vector 1220 of the second object, and the attribute-specific feature vector 1230 of the third object, the third object 1230 is the third object. The one-attribute feature vector 1231 can be exchanged for the representative feature vector 1241.

The image search device 100 can compare the feature vector sequence 1250 including the representative feature vector 1241 with the feature vector sequence of the target image stored in the database to determine the target image corresponding to the search image 1100. The image search device 100 compares the feature vector sequence 1250 including the representative feature vector 1241 with the feature vector sequence of the target image, compares the feature vectors of the attributes of the same category of the same object, and attributes related to each object. Another distance can be calculated, and the target image corresponding to the search result can be determined based on the calculated distance. In one embodiment, the image search device 100 combines the attribute-specific distances related to each object calculated as a result of comparing the feature vector sequence 1250 including the representative feature vector 1241 with the feature vector sequence of the target image, and the combination result. Based on the value, the target image corresponding to the search result can be determined.

In one embodiment, the image search device 100 can transmit the target image selected as the search result to the user terminal 10.

In another embodiment, the image search device 100 extracts a partial image corresponding to the object selected by the user terminal 10 in the target image selected as the search result, and synthesizes the extracted partial image into the search image. As a search result, it can also be transmitted to the user terminal 10. For example, as shown in FIGS. 11 and 13, the image search device 100 replaces the portion including the bag object 1150 in the search image 1100 with the partial images 1311, 1321, 1331 including the bag object in the target image. The combined images 1310, 1320, and 1330 can be transmitted to the user terminal 10 as search result images.

The user can transmit a search image including a plurality of objects to the image search device 100, and easily grasp the coordination style that matches his / her taste through the search result.

FIG. 14 is a block diagram illustrating the configuration of the image search device 100 according to the embodiment.

Referring to FIG. 14, another image retrieval device 100 may include a memory 1410, a database 1430, a communication unit 1450 and a processor 1470 in one embodiment.

Memory 1410 stores at least one program. The memory 1410, the database 1430, the communication unit 1450 and the processor 1470 can be operated by at least one program. In addition, the memory 1410 can further store a generative model for generating a representative feature vector. In addition, the memory 1410 can further store a machine learning model for feature vector extraction.

Database 1430 stores the target image and the feature vector of the target image. The feature vector of the target image is also output by the machine learning model. For example, the feature vector of the target image may include the one output by the dependent model or may include the one output by the independent model. Further, for a target image including a plurality of objects, the feature vector may include a feature vector sequence including the attribute-specific feature vectors of the respective objects. In one embodiment, the database 1430 also comprises a storage device such as a hard disk (HDD) or SSD.

The communication unit 1450 transmits / receives data to / from the user terminal 10. The communication unit 1450 receives the search condition from the user terminal 10 and transmits the search result corresponding to the search condition, for example, the search result image to the user terminal 10.

The processor 1470 acquires a reference feature vector corresponding to a plurality of reference images having the same attribute or assigned the same attribute tag, and has the same attribute or the same attribute from a generation model trained based on the reference feature vector. It is possible to acquire a representative feature vector corresponding to a tag, compare the feature vector of the target image with the representative feature vector, and assign an attribute tag corresponding to the same attribute to at least a part of the target images.

Further, the processor 1470 can select the target image corresponding to the search condition received from the user terminal 10. The processor 1470 selects a target image to which an attribute tag corresponding to the attribute selected or input by the user terminal 10 is assigned, or a feature vector identical / similar to the representative feature vector of the attribute selected or input by the user terminal 10. You can select the target image with.

Further, the processor 1470 can select a target image corresponding to a plurality of search images received from the user terminal 10. Further, the processor 1470 replaces a part of the attribute-specific feature vectors of the search image received from the user terminal 10 with the representative feature vectors of other attributes, and selects the target image based on the replaced result. You can also do it.

On the other hand, the above-described embodiment of the present disclosure can be created in a program executed by a computer, and the created program is also stored in a medium.

The medium is also a continuous storage of computer-executable programs, or a temporary storage for execution or download. The medium is also a variety of recording or storage means in the form of a single piece or a combination of several pieces of hardware, but is not limited to a medium directly connected to a computer system and is distributed on a network. It also does. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tapes; optical recording media such as CD-ROMs (compact disc read only memory) and DVDs (digital versatile discs), and floptical discs. Includes magnetic and optical medium (disk); and ROM (read-only memory), RAM (random access memory), flash memory, etc., and is configured to store program command words. It is also a thing. Further, as an example of other media, a recording medium or a recording medium managed by an app store that distributes applications, a site that supplies or distributes various other software, a server, or the like can be mentioned.

The technical idea of the present disclosure has been described in detail with reference to desirable embodiments, but the technical idea of the present disclosure is not limited to the above-described embodiment and is within the scope of the technical idea of the present disclosure. Therefore, various modifications and changes can be made by those skilled in the art in this field.

(Appendix 1)
In the image search method using the image search device,
At the stage of acquiring reference feature vectors corresponding to multiple reference images containing the same attribute,
From the generated model trained based on the acquired reference feature vector, the stage of acquiring the representative feature vector corresponding to the same attribute, and
A step of comparing the feature vector of the target image stored in the database with the representative feature vector and assigning an attribute tag corresponding to the same attribute to at least a part of the target images of the target images is included. An image search method featuring.
(Appendix 2)
The image search method is
At the stage of receiving attribute selection from the user terminal,
Among the target images, the stage of determining the target image to which the attribute tag corresponding to the selected attribute is assigned, and
The image search method according to Appendix 1, further comprising a step of transmitting the determined target image to the user terminal.
(Appendix 3)
The image search method is
At the stage of receiving attribute selection from the user terminal,
A step of comparing the representative feature vector corresponding to the selected attribute with the feature vector of the target image stored in the database.
The image search method according to Appendix 1, further comprising a step of transmitting a target image selected based on the comparison result to the user terminal among the target images.
(Appendix 4)
The comparison stage is
When the representative feature vector corresponding to the attribute selected by the user terminal is output from the generated model trained based on the reference feature vector output from the dependent model, it corresponds to the selected attribute. Comparing the representative feature vector with the feature vector of the target image output from the dependent model,
When the representative feature vector corresponding to the attribute selected by the user terminal is output from the generated model trained based on the reference feature vector output from the independent model, it corresponds to the selected attribute. The image search method according to Appendix 3, further comprising a step of comparing the representative feature vector and the feature vector of the target image output from the independent model.
(Appendix 5)
The image search method is
At the stage of receiving the search image from the user terminal,
At the stage of acquiring the attribute-specific feature vector of the search image,
Among the attribute-specific feature vectors, the stage of changing the feature vector of the attribute selected by the user terminal to the representative feature vector corresponding to the attribute selected by the user terminal, and
A step of comparing the attribute-specific feature vector including the representative feature vector with the attribute-specific feature vector of the target image stored in the database.
The image search method according to Appendix 1, wherein the target image selected based on the comparison result among the target images stored in the database is transmitted to the user terminal.
(Appendix 6)
The comparison stage is
At the stage of comparing the attribute-specific feature vector including the representative feature vector and the attribute-specific feature vector of the target image between the attributes of the same category and acquiring the attribute-specific distance.
Addendum 5 is characterized by including a step of selecting a target image to be transmitted to the user terminal from the target images stored in the database based on the result of combining the acquired distance sizes for each attribute. The described image search method.
(Appendix 7)
The image search method is
At the stage of receiving a search image containing multiple objects from the user terminal,
The stage of acquiring the feature vector sequence including the attribute-specific feature vectors of each of the plurality of objects included in the search image, and
Among the plurality of objects, the stage of receiving an object selection from the user terminal and
Among the attribute-specific feature vectors of the selected object, the stage of changing the feature vector of the attribute selected by the user terminal to the representative feature vector corresponding to the attribute selected by the user terminal, and
A step of comparing the feature vector sequence including the representative feature vector with the feature vector sequence of the target image stored in the database.
The image search method according to Appendix 1, further comprising a step of transmitting a target image selected based on the comparison result to the user terminal among the target images stored in the database.
(Appendix 8)
The stage of transmitting the selected target image to the user terminal is
Among the plurality of objects included in each of the selected target images, the stage of synthesizing the partial image corresponding to the object selected by the user terminal into the search image, and
The image search method according to Appendix 7, wherein a search image in which the partial images are combined is transmitted to the user terminal as a search result.
(Appendix 9)
In the image search method using the image search device,
At the stage of receiving multiple search images from the user terminal,
The stage of acquiring the feature vectors corresponding to the plurality of search images, and
At the stage of training the generative model with the acquired feature vector,
A step of comparing the representative feature vector output from the generated model trained with the feature vector and the feature vector of the target image stored in the database to determine the target image corresponding to the plurality of search images. When,
An image search method comprising a step of transmitting the determined target image to the user terminal.
(Appendix 10)
The generative model is
The image search method according to Appendix 1 or 9, wherein a GAN (generative adversarial network) including a plurality of FCLs (fully connected layers) is included.
(Appendix 11)
A computer program that causes the computer of the apparatus to execute the image search method according to any one of Supplementary note 1 to 10.
(Appendix 12)
With the processor
A memory for storing at least one program and a generative model,
Contains a database to store the target image, but
The processor is executed by executing the at least one program.
Acquire the reference feature vector corresponding to multiple reference images containing the same attribute.
A representative feature vector corresponding to the same attribute is obtained from the generated model trained based on the obtained reference feature vector.
An image search device characterized by comparing a feature vector of the target image with the representative feature vector and assigning an attribute tag corresponding to the same attribute to at least a part of the target images of the target images.
(Appendix 13)
In the image search method by the user terminal device,
At the stage of connecting to the image search device and selecting attributes,
Among the target images stored in the image search device, the step of receiving the target image to which the attribute tag corresponding to the selected attribute is assigned from the image search device is included.
The target image saved in the image search device is
At least some of the attribute tags corresponding to the plurality of attributes are assigned based on the comparison result between the representative feature vector corresponding to each of the plurality of attributes and the feature vector of the target image. An image search method featuring.

10 User terminal 100 Image search device 300 Generation model 810, 1100 Search image 1310, 1320, 1330 Composite image 1410 Memory 1430 Database 1450 Communication unit 1470 Processor

Claims (8)

In the image search method using the image search device,
The stage of receiving a search image or information related to the search image from the user terminal, and
The stage of receiving the selection for the attribute from the user terminal and
At the stage of acquiring the attribute-specific feature vector of the search image,
Among the attribute-specific feature vectors, the stage of changing the feature vector related to the attribute selected by the user terminal to the representative feature vector corresponding to the attribute selected by the user terminal, and
At the stage of comparing the attribute-specific feature vector including the representative feature vector with the attribute-specific feature vector of the target image stored in the database.
An image search method comprising a step of transmitting a target image selected based on the comparison result to the user terminal among the target images stored in the database. The image search method is
Further including a step of receiving a selection for any one of a plurality of objects included in the search image from the user terminal.
The stage of acquiring the attribute-specific feature vector of the search image is
At the stage of acquiring the attribute-specific feature vectors related to multiple objects in the search image,
Including a step of constructing a feature vector sequence including attribute-specific feature vectors relating to the plurality of objects.
The stage of the change is
Among the attribute-specific feature vectors of the object selected by the user terminal, the step of changing the feature vector associated with the attribute selected by the user terminal to the representative feature vector corresponding to the attribute selected by the user terminal is included. The image search method according to claim 1, which is characterized. The stage of transmitting the selected target image to the user terminal is
Among the plurality of objects included in each of the selected target images, the stage of synthesizing the partial image corresponding to the object selected by the user terminal into the search image, and
The image search method according to claim 2, further comprising a step of transmitting a search image in which the partial images are combined to the user terminal as a search result. In the image search method using the image search device,
At the stage of receiving the first search image and the second search image from the user terminal,
The stage of acquiring the feature vector of the first attribute from the first search image and acquiring the feature vector of the second attribute from the second search image, and
A step of comparing the attribute-specific feature vector including the first attribute feature vector and the second attribute feature vector with the attribute-specific feature vector of the target image stored in the database.
An image search method comprising a step of transmitting a target image selected by the comparison result to the user terminal. The image search method is
At the stage of acquiring reference feature vectors corresponding to multiple reference images containing the same attribute,
The stage of acquiring the representative feature vector corresponding to the same attribute from the generated model trained based on the acquired reference feature vector, and
It further includes a step of comparing the feature vector of the target image stored in the database with the representative feature vector and assigning an attribute tag corresponding to the same attribute to at least a part of the target images among the target images. The image search method according to any one of claims 1 to 4, wherein the image search method is characterized. The comparison stage is
At the stage of comparing the attribute-specific feature vector including the representative feature vector and the attribute-specific feature vector of the target image between the attributes of the same category and acquiring the attribute-specific distance.
Claim 1 is characterized by including a step of selecting a target image to be transmitted to the user terminal from the target images stored in the database based on the result of combining the acquired distance sizes for each attribute. The image search method according to any one of 4 to 4. A program for executing the image search method according to any one of claims 1 to 6. With the processor
Memory to store at least one program,
Contains a database to store the target image, but
The processor is executed by executing the at least one program.
Receive a search image or information related to the search image from the user terminal,
Acquire the attribute-specific feature vector of the search image and
Among the attribute-specific feature vectors, the feature vector related to the attribute selected by the user terminal is changed to the representative feature vector corresponding to the attribute selected by the user terminal.
The attribute-specific feature vector including the representative feature vector is compared with the attribute-specific feature vector of the target image stored in the database.
An image search device characterized in that a target image selected based on the comparison result among the target images stored in the database is transmitted to the user terminal.

Images