JP7211775B2 - Search program, server device, search method and search system - Google Patents

Description

The present invention relates to a search program, a server device, a search method, and a search system.

Conventionally, various search methods have been proposed for searching for similar images similar to a query image (for example, Patent Documents 1 and 2 listed below).

Japanese Patent Application Laid-Open No. 2001-052175 Japanese Patent Application Laid-Open No. 2001-319232

However, in the case of the above search method, both the search target article and the non-search target are included in the query image, and the feature amount of the non-search target is higher in similarity than the feature amount of the search target article. There is a problem that an appropriate image cannot be retrieved when the effect is large.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to output search results according to a search target in image search using a query image.

According to one embodiment of the present invention, the search program has, for example, the following configuration. Namely
an input step of inputting a query image;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images Then, the computer is caused to execute a search step of searching for an image that includes an article similar to the search target and does not include the non-search target from among the plurality of images.

According to the present invention, it is possible to output search results according to a search target in image search using a query image.

It is a figure which shows an example of the system configuration|structure of a search system. It is a figure which shows an example of the hardware constitutions of a server apparatus. It is a figure which shows the specific example of the registration information in a registration information storage part. FIG. 4 is a diagram showing a specific example of feature vector information (product) stored in a feature vector information (product) storage unit; FIG. 4 is a diagram showing a specific example of feature vector information (person) and representative vectors stored in a feature vector information (person) storage unit; It is a figure which shows an example of a functional structure of a product information registration part. It is a figure which shows an example of a functional structure of a product information search part. FIG. 10 is a sequence diagram showing the flow of product information registration processing in the search system; FIG. 10 is a sequence diagram showing the flow of product information search processing in the search system; FIG. 4 is a diagram for explaining learning processing of a feature vector calculation model; 10 is a flow chart showing the flow of feature vector information (person) and representative vector generation processing. FIG. 10 is a first diagram for explaining the feature vector information (person) and processing contents of representative vector generation processing; 10 is a flow chart showing the flow of feature vector information (product) generation processing. FIG. 10 is a diagram for explaining the processing contents of feature vector information (product) generation processing; It is a flowchart which shows the flow of product information search processing. It is a figure for demonstrating the processing content of merchandise information search processing. It is a figure which shows the specific example of product information search processing. FIG. 10 is a first diagram showing an example of search results displayed on the search screen of the user terminal; FIG. 9 is a second diagram showing an example of search results displayed on the search screen of the user terminal; FIG. 10 is a second diagram for explaining the feature vector information (person) and processing contents of representative vector generation processing; It is a figure which shows an example of the chat screen of a user terminal.

The details of each embodiment will be described below. In describing the specifications and drawings according to each embodiment, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

[First embodiment]
<1. System configuration of search system>
First, the system configuration of the search system will be explained. FIG. 1 is a diagram showing an example of the system configuration of a search system.

As shown in FIG. 1, the search system 100 has a server device 110 and user terminals 170_1 to 170_n used by users 180_1 to 180_n, respectively. The user terminals 170_1 to 170_n are connected to the server device 110 via the network 190 so as to be able to communicate therewith.

A flea market service program is installed in the server device 110 , and the server device 110 functions as a flea market service section 120 by executing the program.

The flea market service unit 120 provides users 180_1 to 180_n of user terminals 170_1 to 170_n connected via the network 190 with a place to trade various products.

Here, it is assumed that the flea market service program includes at least a registration program and a search program. By executing these programs, server device 110 functions as flea market service section 120 having product information registration section 130 and product information search section 140 .

When the users 180_1 to 180_n send product information from the user terminals 170_1 to 170_n, the product information registration unit 130 receives the product information and stores it in the registration information storage unit 150 . In addition, in this embodiment, a plurality of items of product information stored in the registered information storage unit 150 are collectively referred to as "registered information".

In addition, the product information registration unit 130 extracts a “feature vector” (a set of a feature amount item and a feature amount value extracted from the image) from the product image included in each stored product information, and extracts the feature vector It saves in the information (product) saving unit 151 .

Furthermore, the product information registration unit 130 registers a specific product image (for example, a product image including a person wearing the product (“wearing image”) other than the product) among the product images included in each item of stored product information. A feature vector is extracted from and stored in the feature vector information (person) storage unit 152 .

When the product information search unit 140 receives a search request including a query image from the user terminals 170_1 to 170_n, the product information search unit 140 uses the feature vector stored in the feature vector information (person) storage unit 152 to search for the person included in the query image. A process for suppressing the influence of the feature amount of is performed. After that, the product information search unit 140 searches for product images including products similar to the search target included in the query image.

Specifically, the product information search unit 140 obtains a feature vector similar to the feature vector of the post-processing query image from among the feature vectors stored in the feature vector information (product) storage unit 151, thereby obtaining a query image. Search for product images that contain products similar to the search target in the image.

The product information search unit 140 transmits the search result including the searched product image to the user terminal (one of the user terminals 170_1 to 170_n) that sent the search request.

<Hardware Configuration of Server Device>
Next, the hardware configuration of the server device 110 included in the search system 100 will be described. FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device;

As shown in FIG. 2 , the server device 110 has a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 and a RAM (Random Access Memory) 203 . The server device 110 also has a GPU (Graphics Processing Unit) 204 . Note that processors (processing circuits, processing circuits) such as the CPU 201 and GPU 204 and memories such as the ROM 202 and RAM 203 form a so-called computer.

Further, the server device 110 has an auxiliary storage device 205 , a display device 206 , an operation device 207 , an I/F (Interface) device 208 and a drive device 209 . Each piece of hardware of the server device 110 is interconnected via a bus 210 . Further, each hardware shown in FIG. 2 is merely an example, and it goes without saying that the server apparatus 110 may be configured by partially changing (for example, removing the GPU 204).

The CPU 201 is an arithmetic device that executes various programs (for example, a flea market service program, etc.) installed in the auxiliary storage device 205 .

A ROM 202 is a non-volatile memory and functions as a main memory. The ROM 202 stores various programs, data, etc. necessary for the CPU 201 to execute various programs installed in the auxiliary storage device 205 . Specifically, the ROM 202 stores boot programs such as BIOS (Basic Input/Output System) and EFI (Extensible Firmware Interface).

A RAM 203 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), and functions as a main storage device. The RAM 203 provides a work area that is expanded when various programs installed in the auxiliary storage device 205 are executed by the CPU 201 .

The GPU 204 is an arithmetic device for image processing, and when the CPU 201 executes the flea market service program, the GPU 204 performs high-speed arithmetic through parallel processing on various image data. Note that the GPU 204 is equipped with an internal memory (GPU memory), and temporarily holds information necessary for parallel processing of various image data.

The auxiliary storage device 205 stores various programs and various data used when the various programs are executed by the CPU 201 . For example, the registration information storage unit 150 , the feature vector information (product) storage unit 151 , and the feature vector information (person) storage unit 152 are implemented in the auxiliary storage device 205 .

The display device 206 is a display device that displays the internal state of the server device 110 . The operation device 207 is an input device used when an administrator of the server device 110 inputs various instructions to the server device 110 . The I/F device 208 is a connection device for connecting to the network 190 and communicating with the user terminals 170_1 to 170_n.

A drive device 209 is a device for setting a recording medium 220 . The recording medium 220 here includes media for optically, electrically or magnetically recording information, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 220 may also include a semiconductor memory or the like that electrically records information, such as a ROM or flash memory.

Various programs to be installed in the auxiliary storage device 205 are installed by, for example, setting the distributed recording medium 220 in the drive device 209 and reading the various programs recorded in the recording medium 220 by the drive device 209. be done. Alternatively, various programs installed in the auxiliary storage device 205 may be installed by being downloaded via the network 190 .

<Specific example of registration information>
Next, a specific example of registration information in the registration information storage unit 150 will be described. FIG. 3 is a diagram showing a specific example of registration information in a registration information storage unit.

As shown in FIG. 3, the registration information 300 is composed of a plurality of item information. Specifically, the registration information 300 includes "product ID", "product attribute information", and "product image" as information items. The registration information 300 is configured by being stored in the registration information storage unit 150 .

For example, the "product ID" stores an identifier for identifying the product among the product information. "Product attribute information" stores information indicating product attributes (for example, product category, product brand, product title, etc.) among product information. "Product image" stores the product image of the product information.

The product images stored in the "product image" include not only the image of the product itself, but also the product image (wearing image) including the person wearing the product.

<Specific example of feature vector information (product)>
Next, a specific example of the feature vector information (product) stored in the feature vector information (product) storage unit 151 will be described. The feature vector information (product) storage unit 151 stores the feature vector extracted from the product image as feature vector information (product) in association with the product information. FIG. 4 is a diagram showing a specific example of feature vector information (product) stored in a feature vector information (product) storage unit.

As shown in FIG. 4, the feature vector information (product) 400 includes, as information items, "product ID", "product attribute information", and "product image", which are information items related to product information. feature vector". "Feature vector" stores a feature vector extracted from the product image stored in "product image".

As described above, a feature vector is a set of a feature amount item and a feature amount value extracted from an image. In the example of FIG. " is extracted. Also, in the example of FIG. 4, the value of the feature amount is shown as the length of the bar graph.

<Specific example of feature vector information (person)>
Next, a specific example of the feature vector information (person) stored in the feature vector information (person) storage unit 152 will be described. The feature vector information (person) storage unit 152 associates the feature vector extracted from the wearing image with the product information and stores it as feature vector information (person), based on the feature vector extracted from the wearing image. Save the calculated representative vector.

FIG. 5 is a diagram showing a specific example of the feature vector information (person) and the representative vector stored in the feature vector information (person) storage unit.

As shown in FIG. 5, the feature vector information (person) 500 includes "product ID", "product attribute information", "wearing image", and "feature vector" as information items. In the case of the feature vector information (person) 500, a wearing image selected from product images included in each product information constituting the registration information 300 is stored in the "wearing image", and the corresponding product identifier and product attribute are stored. Information is stored in "product ID" and "product attribute information", respectively. Further, the "feature vector" stores a feature vector extracted from the wearing image stored in the "wearing image". As in FIG. 4, the example of FIG. 5 indicates that "feature 1" to "feature m" have been extracted as feature amount items. Also, in the example of FIG. 5, the value of the feature amount is shown as the length of the bar graph.

The representative vector 510 is calculated based on the feature vector of each wearing image stored in the “feature vector” of the feature vector information (person) 500 . The representative vector 510 is obtained, for example, by calculating the average value of the feature vectors of the wearing images. Alternatively, the representative vector 510 is obtained by calculating the median value of the feature vectors of the wearing images. Alternatively, the representative vector 510 is obtained by integerizing the feature vector of each wearing image and calculating the mode.

<Functional configuration of the product information registration unit>
Next, the functional configuration of the product information registration unit 130 included in the flea market service unit 120 of the server device 110 will be described. FIG. 6 is a diagram illustrating an example of a functional configuration of a product information registration unit; As shown in FIG. 6 , product information registration unit 130 has product information acquisition unit 610 , product information (wearing image) selection unit 620 , image feature extraction unit 630 , person feature calculation unit 640 , and product information readout unit 650 .

When the user terminals 170_1 to 170_n send product information from the user terminals 170_1 to 170_n in order for the users 180_1 to 180_n to exhibit products, the product information acquisition unit 610 stores the product information in the registration information storage unit 150 each time it receives the product information. Thereby, the registration information 300 in the registration information storage unit 150 is updated each time the product information acquisition unit 610 receives the product information.

Product information (wearing image) selection unit 620 selects and reads product information including a wearing image from among the product information constituting registration information 300 in registration information storage unit 150 , and notifies image feature extraction unit 630 of the product information. . The product information (wearing image) selection unit 620 selects the product image including the wearing image, for example, at the timing when the product information including the wearing image is registered as the registration information 300 in the registration information storage unit 150 to a certain extent. shall be read as

In this case, the image feature extraction unit 630 extracts a feature vector from the wearing image included in the product information notified by the product information (wearing image) selection unit 620, associates it with the product information, and outputs it to the person feature calculation unit 640. Notice.

Personal characteristic calculation section 640 calculates representative vector 510 based on the characteristic vector associated with each product information notified from personal characteristic calculation section 640 . Further, the person feature calculation unit 640 stores the product information associated with the feature vector extracted from the wearing image as the feature vector information (person) 500 together with the representative vector 510 in the feature vector information (person) storage unit 152 . do.

The product information reading unit 650 reads the registration information 300 in the registration information storage unit 150 and notifies the image feature extraction unit 630 of it.

In this case, the image feature extractor 630 extracts a feature vector from the product image included in each product information constituting the registration information 300 notified from the product information readout unit 650 . The image feature extraction unit 630 also associates the extracted feature vector with the product information, and stores the feature vector information (product) 400 in the feature vector information (product) storage unit 151 .

<Functional configuration of the product information search unit>
Next, the functional configuration of the product information search section 140 included in the flea market service section 120 of the server device 110 will be described. FIG. 7 is a diagram illustrating an example of a functional configuration of a product information search unit; As shown in FIG. 7, the product information search unit 140 includes a query image acquisition unit 710, a search object detection unit 720, an image feature extraction unit 630, an image feature conversion unit 740, a similar vector calculation unit 750, and a search result output unit 760. have

The query image acquisition unit 710 is an example of an input unit, receives search requests from the user terminals 170_1 to 170_n, and acquires query images included in the received search requests.

The search target object detection unit 720 detects a region including a product to be searched from the query image as a search target region. However, the function of the search object detection unit 720 is not essential, and the product information search unit 140 does not have to execute the search object detection unit 720 .

The image feature extraction unit 630 extracts a feature vector from the search target area detected by the search target object detection unit 720 and notifies the image feature conversion unit 740 of the feature vector.

The image feature conversion unit 740 reads out the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector notified from the image feature extraction unit 630. Calculate the feature vector of . Specifically, the image feature conversion unit 740 subtracts the value of the feature amount of the representative vector 510 from the value of the feature amount of the feature vector for each item of the feature amount, thereby calculating the feature vector after the subtraction. As a result, the feature vector after the subtraction becomes a feature vector in which the influence of the person's feature amount is suppressed.

Further, the image feature conversion unit 740 converts the feature vector after subtraction by zeroing the negative values included in the feature vector after subtraction, and calculates the feature vector after conversion. Specifically, the image feature transforming unit 740 transforms the value of the feature amount into a zero value (non-negative) for the item of the feature amount that has become a negative value due to the subtraction among the items of each feature amount. As a result, the feature vector after conversion becomes a feature vector that does not contain negative values. Although the subtraction is performed to suppress the influence of the person's feature amount, if the effect of the person's feature amount appears as a negative value as a result of the subtraction, it does not mean that the influence of the person's feature amount has been suppressed. , the image feature conversion unit 740 appropriately suppresses the influence of the person's feature amount by converting to a zero value.

The similarity vector calculation unit 750 is an example of a search unit, reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151, and finds similarity to the converted feature vector notified from the image feature conversion unit 740. Find feature vectors that Further, the similarity vector calculation unit 750 notifies the searched feature vector to the search result output unit 760 .

The search result output unit 760 sends the product information associated with the feature vector notified from the similarity vector calculation unit 750 as the search result to the user terminal (any of the user terminals 170_1 to 170_n) that sent the search request. Send.

<Product information registration process flow>
Next, the flow of product information registration processing in the search system 100 will be described. FIG. 8 is a sequence diagram showing the flow of product information registration processing in the search system.

In step S801, the image feature extraction unit 630 included in the product information registration unit 130 of the server device 110 uses a learning model (feature vector calculation model; details will be described later) for machine learning to extract feature vectors. Get training images. As the learning image, for example, a product image or the like included in a part of product information of a plurality of product information constituting the registration information 300 in the registration information storage unit 150 is used. Note that images to be used for learning images are not limited to product images, and may be any image that can be downloaded on the network 190, for example.

In step S802, the image feature extraction unit 630 machine-learns a feature vector calculation model using the acquired learning image.

In steps S810_1 to S810_3, the user terminals 170_1 to 170_n accept input of product information about products to be exhibited from the users 180_1 to 180_n, and transmit the input to the server device 110. FIG.

In steps S811_1 to S811_3, the product information acquisition unit 610 included in the product information registration unit 130 of the server device 110 sequentially stores the received product information (product information on products exhibited by the users 180_1 to 180_n) as registered information. Stored in unit 150 . Thereby, the registration information 300 is configured in the registration information storage unit 150 .

In step S820, product information (wearing image) selection unit 620, image feature extraction unit 630, and person feature calculation unit 640 included in product information registration unit 130 of server device 110 perform feature vector information (person) and representative vector generation processing. to run. As a result, the feature vector information (person) storage unit 152 stores the feature vector information (person) 500 and the representative vector 510 .

In step S830_1, the product information reading unit 650 and the image feature extraction unit 630 execute feature vector information (product) generation processing. As a result, the feature vector information (product) 400 is stored in the feature vector information (product) storage unit 151 .

In steps S810_4 to S810_6, the user terminals 170_1 to 170_n accept input of product information from the users 180_1 to 180_n and transmit it to the server device 110. FIG.

In steps S811_4 to S811_6, the product information acquisition unit 610 included in the product information registration unit 130 of the server device 110 sequentially stores the received product information in the registered information storage unit 150. FIG. Thereby, the registration information 300 in the registration information storage unit 150 is updated.

In step S830_2, the product information reading unit 650 and the image feature extraction unit 630 execute feature vector information (product) generation processing. As a result, the feature vector information (product) 400 in the feature vector information (product) storage unit 151 is updated. Thereafter, similar processing is repeated in steps S810_7 to S810_9 and steps S811_7 to S811_9.

In this way, the retrieval system 100 first executes the feature vector information (person) and representative vector generation process (step S820) once to generate the representative vector 510. FIG. Thereafter, in the search system 100, each time the registration information 300 is updated multiple times (or each time a predetermined period of time elapses), the feature vector information (product) generation process is executed to update the feature vector information (product) 400. do.

<Product information search process flow>
Next, the flow of product information search processing in the search system 100 will be described. FIG. 9 is a sequence diagram showing the flow of product information search processing in the search system.

In step S900_1, the user terminals 170_1-170_n receive search requests from the users 180_1-180_n and transmit them to the server device 110. FIG.

In step S910_1, the product information search unit 140 of the server device 110 executes product information search processing, and transmits the search result to the user terminal (one of the user terminals 170_1 to 170_n) that sent the search request.

In step S901_1, the user terminals 170_1-170_n receive and display the search results. Thereafter, in the search system 100, every time the user terminals 170_1 to 170_n receive a search request, the server device 110 executes product information search processing and displays the search results on the user terminals 170_1 to 170_n.

Note that the product information search processing shown in FIG. 9 is executed in parallel with the processing after step S810_1 of the product information registration processing shown in FIG.

<Details of each step of the product information registration process>
Next, each process included in the product information registration process shown in FIG. 8 (here, feature vector calculation model learning process, feature vector information (person) and representative vector generation process, feature vector information (product) generation process) will be described in detail.

(1) Details of Feature Vector Calculation Model Learning Processing (Step S802) First, details of the feature vector calculation model learning processing (Step S802) will be described. FIG. 10 is a diagram for explaining the learning process of the feature vector calculation model. The image feature extraction unit 630 reads out the learning image and inputs it to the feature vector calculation model 1000 . Note that when a product image included in some product images is used as the learning image, the image feature extraction unit 630 reads out the product information from the registered information storage unit 150 as the learning image before the learning process. , is input to the feature vector calculation model 1000 .

Product attribute information such as a corresponding category, brand, pattern, etc., is associated in advance with the learning image. Perform machine learning of the model parameters so that they are output. Note that the feature vector calculation model 1000 is assumed to be configured by a neural network (for example, a convolutional neural network) or the like.

As a result, the image feature extraction unit 630 generates a trained model for the feature vector calculation model 1000 . The image feature extraction unit 630 extracts feature vectors using a learned model of the feature vector calculation model 1000 .

(2) Details of Feature Vector Information (Person) and Representative Vector Generation Processing (Step S820) Next, details of the feature vector information (person) and representative vector generation processing (Step S820) will be described. FIG. 11 is a flowchart showing the flow of feature vector information (person) and representative vector generation processing.

As shown in FIG. 11, in step S1101, the product information (wearing image) selection unit 620 selects product information including a wearing image from the product information constituting the registered information 300 in the registered information storage unit 150. Read out and acquire the wearing image.

In step S1102, the image feature extraction unit 630 extracts feature vectors using the learned model of the feature vector calculation model 1000. FIG. FIG. 12 is a first diagram for explaining processing contents of feature vector information (person) and representative vector generation processing.

As shown in FIG. 12 , the image feature extraction unit 630 sequentially extracts the worn image from the learned model 1200 (a learned model in which up to the k-th layer of the learned model for the feature vector calculation model 1000 can be executed). to enter. As a result, the image feature extraction unit 630 sequentially extracts feature vectors from the k-th layer. In the k-th layer, which is the intermediate layer of the trained model, features that are easy for humans to recognize are calculated. Therefore, by visualizing the k-th layer and calculating the feature vectors, extract feature vectors that are easy for humans to recognize. Because you can. It should be noted that the feature vector extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value. When calculating between feature vectors, if a negative value is included in the feature value, there will be a case where the negative value is subtracted. In such a case, the feature value will swing to the positive side. Because it becomes In other words, this is to avoid a situation in which the influence of the person's feature amount is not suppressed despite the subtraction.

In step S<b>1103 , the person feature calculation unit 640 calculates the representative vector 510 . For example, the person feature calculator 640 calculates the representative vector 510 by calculating the average value of each feature vector extracted from each wearing image (see FIG. 12). Note that the method of calculating the representative vector 510 is not limited to this. For example, the feature vector of the product image is subtracted from the feature vector of each wearing image, the negative values are made non-negative, and then the average value is calculated. , the representative vector 510 may be calculated. According to this calculation method, there is an advantage that the product image is not subtracted too much compared to the case where the representative vector is calculated directly from the feature vector of the wearing image.

In step S<b>1104 , the person feature calculation unit 640 generates feature vector information (person) 500 by associating the read product information with the extracted feature vector, and stores the generated feature vector information (person) 500 in the feature vector information (person) storage unit 152 . do. In addition, the person feature calculation unit 640 stores the calculated representative vector in the feature vector information (person) storage unit 152 .

(3) Details of Feature Vector Information (Product) Generation Processing (Step S830_1, etc.) Next, details of the feature vector information (product) generation processing (Step S830_1, etc.) will be described. FIG. 13 is a flowchart showing the flow of feature vector information (product) generation processing.

As shown in FIG. 13, in step S1301, the product information reading unit 650 reads the registered information 300 in the registered information storage unit 150, and acquires product images included in each item of product information.

In step S1302, the image feature extraction unit 630 uses the trained model 1200 to extract feature vectors from the product image. FIG. 14 is a diagram for explaining the processing contents of the feature vector information (product) generation processing.

As shown in FIG. 14, the image feature extraction unit 630 sequentially extracts product images from a trained model 1200 (a trained model in which up to the k-th layer of the trained model for the feature vector calculation model 1000 is made executable). to enter. As a result, the image feature extraction unit 630 sequentially extracts feature vectors 1410_1 to 1410_p from the k-th layer. Note that, as described above, the feature vectors 1410_1 to 1410_p extracted from the k-th layer are configured so that the value of each feature amount is a non-negative value.

In step S<b>1303 , the image feature extraction unit 630 generates feature vector information (product) 400 by associating the read product information with the extracted feature vector, and stores the generated feature vector information (product) 400 in the feature vector information (product) storage unit 151 . do.

<Details of product information search processing>
Next, details of the product information search processing (step S910_1, etc.) will be described. FIG. 15 is a flow chart showing the flow of product information search processing. As shown in FIG. 15, in step S1501, the query image acquisition unit 710 receives search requests from the user terminals 170_1 to 170_n and acquires query images included in the received search requests.

In step S<b>1502 , the search target object detection unit 720 detects a region including the product to be searched as a search target region from the acquired query image.

In step S1503, the image feature extraction unit 630 extracts feature vectors from the search target area of the acquired query image. FIG. 16 is a diagram for explaining the details of the product information search process. As shown in FIG. 16, the image feature extraction unit 630 extracts the search target region detected from the query image by the trained model 1200 (the learned model for the feature vector calculation model 1000, which is made executable up to the k-th layer). trained model). As a result, the image feature extraction unit 630 extracts the feature vector 1600 of the search target area from the k-th layer. Note that, as described above, the feature vector 1600 extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value.

In step S1504, the image feature conversion unit 740 reads the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector 1600 of the search target region extracted in step S1503. do. Thereby, the image feature conversion unit 740 calculates a feature vector after subtraction.

In step S1505, the image feature transforming unit 740 transforms negative feature amounts into zero values (non-negative) in the feature vectors after subtraction. Thereby, the image feature conversion unit 740 calculates a feature vector after conversion.

In step S1506, the similarity vector calculation unit 750 reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151, and searches for a feature vector similar to the post-conversion feature vector.

In step S<b>1507 , the search result output unit 760 identifies product information associated with the feature vector searched by the similarity vector calculation unit 750 in the feature vector information (product) 400 . In addition, the search result output unit 760 transmits the specified product information as a search result to the user terminal (any of the user terminals 170_1 to 170_n) that sent the search request.

<Specific example of product information search processing>
Next, a specific example of product information search processing will be described. FIG. 17 is a diagram showing a specific example of product information search processing. In each feature vector shown in FIG. 17, the horizontal axis indicates the item of the feature amount, and the vertical axis indicates the value of the feature amount.

The image feature conversion unit 740 subtracts the representative vector 510 from the feature vector 1600 of the search target area to calculate a feature vector 1701 after subtraction. As a result, in the feature vector 1701 after subtraction, the feature amount of the search target is weighted more than the feature amount of the non-search target (the influence of the feature amount of the person included in the query image can be suppressed). .

Here, in the image feature conversion unit 740, since the value of the feature amount is subtracted for each feature amount item, the value of the feature amount after subtraction may be a negative value depending on the item of the feature amount ( See dotted line area 1702).

The image feature conversion unit 740 converts the value of the feature amount to zero for the item whose feature amount value is a negative value in the feature vector 1701 after the subtraction (see the dotted line area 1704). Accordingly, the image feature conversion unit 740 calculates a feature vector 1703 after conversion.

The similarity vector calculation unit 750 selects a product having a high degree of similarity with the post-conversion feature vector 1703 from among the product image feature vectors 1410_1 to 1410_p included in each product information read from the feature vector information (product) storage unit 151. Find the feature vector of an image.

Note that the similarity vector calculation unit 750 may search for feature vectors of a plurality of product images whose similarities are equal to or greater than a predetermined threshold, or may search for the feature vector of the product image with the highest similarity. Due to space limitations, the example in FIG. 17 shows a state in which the feature vector 1410_i of the product image with the highest degree of similarity is retrieved.

The search result output unit 760 reads the product information 1710 associated with the product image feature vector 1410 — i from the feature vector information (product) storage unit 151 . In addition, the search result output unit 760 transmits the read product information 1710 as a search result to the user terminal (one of the user terminals 170_1 to 170_n) that sent the search request.

<Specific example of search result of product information search process>
Next, specific examples of search results displayed on the search screens of the user terminals 170_1 to 170_n will be described. FIG. 18 is a first diagram showing an example of search results displayed on the search screen of the user terminal.

As shown in FIG. 18, the search screen 1800 includes a "read query image" button 1811 for reading a query image and a query image display field 1801 for displaying the read query image. The search screen 1800 also includes a "search request" button 1812 for sending a search request including a query image, and a search result display field for displaying search results sent from the server apparatus 110 in response to the search request. 1821 are included.

FIG. 18(a) shows the search result (here, 15 product images) when the product information search process (see FIG. 15) is executed by the product information search unit 140 in the server device 110. showing the situation.

As shown in FIG. 18A, according to the product information search processing by the product information search unit 140, even if the query image includes clothes and a person wearing the clothes, It is possible to search for product images that include clothes that do not include people.

On the other hand, FIG. 18B shows, as a comparative example, how search results are displayed when general product information search processing is executed.

As shown in FIG. 18B, according to general product information retrieval processing, when a query image includes clothes and a person wearing the clothes, product images including the person are retrieved. will be

That is, in general product information search processing, product images similar to non-search targets included in the query image are searched.

On the other hand, according to the product information search processing by the product information search unit 140, as described above, the non-search target included in the query image includes products similar to the search target included in the query image. You can search for product images that do not contain.

FIG. 19 is a second diagram showing an example of search results displayed on the search screen of the user terminal. Since the configuration within search screen 1800 is the same as the configuration within search screen 1800 shown in FIG. 18, description of the configuration within search screen 1800 is omitted here.

A difference from FIG. 18 is the query image displayed in the query image display field 1801 . In the case of the query image displayed in the query image display field 1801 of FIG. 19, a pattern is reflected in the background of the person wearing clothes.

As shown in FIG. 19(a), according to the product information search unit 140, the query image includes the clothes to be searched and the person wearing the clothes, as in FIG. 18(a). Even in this case, it is possible to search for product images that include clothing similar to the clothing in question and that do not include a person. In addition, in the case of FIG. 19A, even if a pattern is included in the background of a person wearing clothes, no product image similar to the background pattern is retrieved. This is because the search target object detection unit 720 detects an area including the product to be searched as a search target area. Note that the rectangular area in the query image displayed in the query image display field 1801 indicates the search target area detected by the search target object detection unit 720 .

On the other hand, FIG. 19B shows, as a comparative example, a state in which search results are displayed when the search target detection unit 720 does not execute processing for detecting an area including a search target product as a search target area. is shown.

As shown in FIG. 19B, if the search target area detection process is not executed, product images similar to areas other than the search target area included in the query image are retrieved.

That is, when the process of detecting the search target area is not executed, product images similar to the non-search target included in the query image are searched.

On the other hand, according to the product information search processing by the product information search unit 140, as described above, the non-search target included in the query image includes products similar to the search target included in the query image. You can search for product images that do not contain.

<Summary>
As is clear from the above description, in the server device according to the first embodiment,
・Receive a search request sent from a user terminal and acquire a query image.
・If the query image contains both a search target (e.g., clothes) and a non-search target (e.g., person), among multiple product images, include products similar to the search target, and Search for product images that do not include non-search targets.

Thus, according to the server device according to the first embodiment, it is possible to output search results corresponding to search targets in product searches using query images.

[Second embodiment]
In the above-described first embodiment, product information including wearing images is selected from the registered information 300, and the wearing images are sequentially input to the trained model 1200 to calculate the representative vector from the extracted feature vectors. explained as. However, the method of calculating representative vectors is not limited to this.

For example, when a product image including a wearing image is selected, the selected product image may be classified by category, representative vector candidates may be extracted for each classified category, and then the representative vector may be calculated.

FIG. 20 is a second diagram for explaining the processing contents of the feature vector information (person) generation processing. The difference from FIG. 12 is that in the case of FIG. 20, wearing images are classified by category and sequentially input to the trained model 1200 for each category.

As a result, for example, category 1 feature vectors are extracted for wearing images classified into category 1, and category 2 feature vectors are extracted for wearing images classified into category 2. Furthermore, for wearing images classified into category x, feature vectors of category x are extracted.

Here, in the second embodiment, the person feature calculation unit 640 calculates representative vector candidates for each category based on feature vectors extracted for each category. Furthermore, the person characteristic calculation unit 640 acquires representative vector candidates for each category and calculates a representative vector 2000 by calculating an average value.

However, the method of calculating the representative vector 2000 is not limited to this. For example, for each category, the feature vector of the product image is subtracted from the feature vector of each wearing image, the negative values are non-negative, and then the average value is calculated between the categories to calculate the representative vector 2000. good.

As is clear from the above description, in the server device according to the second embodiment, when calculating the representative vector based on the wearing image, the representative vector candidate is calculated according to the category, and then the representative vector is calculated. .

As a result, according to the server device according to the second embodiment, it is possible to obtain the same effects as those of the first embodiment, and to further optimize the representative vector.

[Third embodiment]
In the first and second embodiments, the search screen 1800 is displayed on the user terminals 170_1 to 170_n when executing the product information search process. However, the product information search process may be executed via a screen other than the search screen 1800 (for example, a chat screen).

FIG. 21 is a diagram illustrating an example of a chat screen of a user terminal; In FIG. 21, the length of the screen in the vertical direction is increased in order to fit the display contents displayed by scrolling in the vertical direction into one figure.

As shown in FIG. 21, by inputting a query image 2101 on the chat screen, a question 2102 ("Do you want to search by image?") is displayed. ). As a result, a search request including the query image 2101 is transmitted to the server device 110 , and the server device 110 executes product information search processing, thereby displaying a search result 2104 on the chat screen 2100 .

Thus, according to the chat screen 2100, the users 180_1 to 180_n can send search requests via chat instead of pressing the "search request" button.

[Other embodiments]
In the first and second embodiments described above, each section (query image acquisition section 710 to search result output section 760) of the product information search section 140 is provided on the server device 110 side. However, some functions of the product information search unit 140 may be provided on the user terminals 170_1 to 170_n. For example, the query image acquisition unit 710, the search object detection unit 720, and the image feature extraction unit 630 may be provided on the user terminals 170_1 to 170_n.

As a result, the server apparatus 110 can receive feature vectors from the user terminals 170_1 to 170_n and execute product information search processing. However, in this case, the user terminals 170_1 to 170_n must acquire the learned model 1200 in advance.

Alternatively, the query image acquisition unit 710, search object detection unit 720, image feature extraction unit 630, and image feature conversion unit 740 may be provided on the user terminals 170_1 to 170_n side. As a result, the server apparatus 110 can receive the converted feature vectors from the user terminals 170_1 to 170_n and execute the product information search process. However, in this case, the user terminals 170_1 to 170_n must acquire the trained model 1200 and the representative vector 510 in advance.

Alternatively, all functions of the product information search unit 140 may be provided on the user terminals 170_1 to 170_n. This allows the user terminals 170_1 to 170_n to execute product information search processing. However, in this case, the user terminals 170_1 to 170_n need to acquire the learned model 1200 and the representative vector 510 in advance, and acquire the feature vector information (product) 400 periodically. This is effective when the number of products is not large or when the frequency of product updates is low.

Further, in the first and second embodiments, the feature vector information (person) and representative vector generation processing is executed only once, and the feature vector information (product) generation processing is executed, for example, every time a predetermined time elapses. was explained as being executed on However, the feature vector information (person) and representative vector generation processing may also be configured to be executed each time a predetermined time elapses, similarly to the feature vector information (merchandise) generation processing. Further, the feature vector information (product) generation process may be configured to be executed each time product information is stored and registration information is updated.

Further, in the first and second embodiments described above, the search target is clothes, and the non-search target is the person who wears the clothes. Then, a representative vector is calculated for a wearing image in which the person wears the clothes. However, the search target is not limited to clothing, and may be accessories, accessories, and the like. In this case, the non-search target is the person wearing the accessory or the person holding the accessory. Then, a representative vector is calculated for a wearing image in which the person wears the accessory (or holds the accessory).

In other words, the wearing image includes all images including a person photographed together with the search target, such as wearing clothes, wearing accessories, holding small items, and the like.

Non-search targets are not limited to people. For example, in an image in which a product is placed on a chair, a chair may be a non-search target. Alternatively, in an image in which a product is hung on a hanger, the hanger may be set as a non-search target. However, if these are not to be searched, it is necessary to calculate a representative vector based on an image containing them.

Further, in the first and second embodiments described above, the product ID, product attribute information, product image, etc. are transmitted as search results. You may choose to send one. Information other than the product ID, product attribute information, and product image may be transmitted as the search result.

Further, in the first embodiment, even if the query image includes clothes and a person wearing the clothes, the product information search unit 140 can find clothes similar to the clothes, In addition, it was shown that it is possible to retrieve product images that do not include people. However, the effect of the product information search processing by the product information search unit 140 is not limited to this. Even when the query image contains only clothes (that is, when the query image is not a wearing image), the product information search unit 140 finds a product image that contains clothes similar to the clothes and does not contain a person. It goes without saying that you can search for

In addition, in the first and second embodiments, the product information search unit 140 calculates the post-conversion feature vector so as to suppress the influence of the person's feature amount on the query image. However, the method of calculating the feature vector after conversion is not limited to this, and for example, the feature vector after conversion may be calculated so that the user's tastes and preferences are clearer in the query image. This makes it possible to implement product recommendation processing for the user.

Further, in the first and second embodiments, the case where the target image searched by the product information search unit 140 is a product image has been described. However, the target images searched by the product information search unit 140 are not limited to product images. For example, it may be a scene image, a painting image, or the like. However, in the case of a scene image, for example, in the query image, the post-conversion feature vector is calculated so as to suppress the influence of the feature amounts of utility poles and automobiles. In the case of a painting image, for example, in the query image, the post-conversion feature vector is calculated so as to suppress the influence of the feature amount of the frame shape.

Further, in the first and second embodiments, the case of searching for product images that include products similar to the search target and that do not include non-search targets has been described. However, images to be searched are not limited to product images, and for example, any image that includes an article similar to the search target and does not include non-search targets may be searched.

It should be noted that the present invention is not limited to the configurations shown here, such as combinations with other elements, to the configurations listed in the above embodiments. These points can be changed without departing from the gist of the present invention, and can be determined appropriately according to the application form.

100: Search system 110: Server device 120: Flea market service unit 130: Product information registration unit 140: Product information search unit 150: Registered information storage unit 151: Feature vector information (product) storage unit 152: Feature vector information (person) Storage units 170_1 to 170_n: User terminal 300: Registration information 400: Feature vector information (product)
500: Feature vector information (person)
510 : representative vector 610 : product information acquisition unit 620 : product information (wearing image) selection unit 630 : image feature extraction unit 640 : person feature calculation unit 650 : product information reading unit 710 : query image acquisition unit 720 : search target object detection Unit 740: Image feature conversion unit 750: Similarity vector calculation unit 760: Search result output unit 1000: Feature vector calculation model 1200: Trained models 1410_1 to 1410_p: Product image feature vector 1600: Search target region feature vector 1701: Feature vector 1703 after subtraction: Feature vector 1800 after conversion: Search screen 2000: Representative vector

Claims (9)

an input step of inputting a query image;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a search program for causing a computer to execute a search step of searching, from among a plurality of images, an image containing an article similar to the search target and not containing the non-search target. The search step includes:
When the query image includes a search target and a person photographed together with the search target, a feature amount obtained by removing the feature amount of the person from the feature amount of the query image and the feature amount of a plurality of images. 2. The search program according to claim 1 , wherein an image including an article similar to the search target and not including a person is searched from among the plurality of images based on . The search step includes:
3. The search program according to claim 2, wherein a search is performed by weighting the feature amount of a search target included in the feature amount of the query image rather than the feature amount of a person included in the feature amount of the query image. The search step includes:
Subtracting the feature vector containing the feature amount of the query image by the feature vector containing the feature amount of the image containing the article and the person photographed with the article,
Non-negative the negative feature values included in the subtracted feature vector,
3. The search program according to claim 2, wherein a feature vector obtained by non-negating the feature quantity of the negative value is used to calculate a similarity with each feature vector including the feature quantity of each of the plurality of images. causing a computer to perform a calculation step of calculating a representative vector based on each feature vector including feature amounts of each of a plurality of images including an article and a person photographed with the article;
The search step includes:
5. The search program according to claim 4, wherein the representative vector is subtracted from the feature vector containing the feature amount of the query image. The calculation step includes:
A plurality of images containing an article and a person photographed with the article are classified according to the category of the article. Calculate representative vector candidates for each
6. The search program according to claim 5, wherein said representative vector is calculated based on representative vector candidates calculated for each category. an input unit for inputting a query image;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images and a search unit that searches a plurality of images for an image that includes an article similar to the search target and does not include the non-search target. an input step of inputting a query image;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images and searching a plurality of images for an image that includes an article similar to the search target and does not include the non-search target. A search system in which a terminal and a server device are communicably connected,
an input unit for inputting a query image;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images and a retrieval unit for retrieving, from among a plurality of images, an image that includes an article similar to the retrieval target and does not include the non-retrieval target.

Images