JP7211776B2 - Search program, server device, search method and search system - Google Patents

Description

The present invention relates to a search program, a server device, a search method, and a search system.

Conventionally, various search methods have been proposed for searching for similar images similar to a query image (for example, Patent Documents 1 and 2 listed below).

Japanese Patent Application Laid-Open No. 2001-052175 Japanese Patent Application Laid-Open No. 2001-319232

However, in the case of the above search method, both the search target article and the non-search target are included in the query image, and the feature amount of the non-search target is higher in similarity than the feature amount of the search target article. There is a problem that an appropriate image cannot be retrieved when the effect is large.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to output search results according to a search target in image search using a query image.

According to one embodiment of the present invention, the search program has, for example, the following configuration. Namely
an acquisition step of acquiring an image of the selected one item as a query image when one item is selected from the listed items and the next screen is displayed;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a retrieving step of retrieving, from among a plurality of images, an image that includes an article similar to the search target and does not include the non-search target;
causing the computer to execute an output step of outputting the searched image as an image of a related article related to the selected one article on the next screen after the transition.

Advantageous Effects of Invention According to the present invention, it is possible to output search results corresponding to a search target in a product search using a query image.

It is a figure which shows an example of the system configuration|structure of a search system. It is a figure which shows an example of the hardware constitutions of a server apparatus. It is a figure which shows the specific example of the registration information in a registration information storage part. FIG. 4 is a diagram showing a specific example of feature vector information (product) stored in a feature vector information (product) storage unit; FIG. 4 is a diagram showing a specific example of feature vector information (person) and representative vectors stored in a feature vector information (person) storage unit; It is a figure which shows an example of a functional structure of a product information registration part. It is a figure which shows an example of the functional structure of a goods display part and a goods information search part. FIG. 10 is a sequence diagram showing the flow of product information registration processing in the search system; FIG. 10 is a sequence diagram showing the flow of product information display/search processing in the search system; FIG. 4 is a diagram for explaining learning processing of a feature vector calculation model; 10 is a flow chart showing the flow of feature vector information (person) and representative vector generation processing. FIG. 10 is a first diagram for explaining the feature vector information (person) and processing contents of representative vector generation processing; 10 is a flow chart showing the flow of feature vector information (product) generation processing. FIG. 10 is a diagram for explaining the processing contents of feature vector information (product) generation processing; It is a flowchart which shows the flow of product list display and product information search processing. It is a figure for demonstrating the processing content of merchandise information search processing. It is a figure which shows the specific example of product information search processing. It is a figure which shows the example of a search result. It is the 1st flowchart which shows the flow of goods selection processing. FIG. 11 is a first diagram showing an example of a product list screen and a product detail screen; FIG. 11 is a second diagram showing an example of a product list screen and a product detail screen; FIG. 11 is a second flowchart showing the flow of product selection processing; FIG.

The details of each embodiment will be described below. In describing the specifications and drawings according to each embodiment, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

[First Embodiment]
<1. System configuration of search system>
First, the system configuration of the search system will be explained. FIG. 1 is a diagram showing an example of the system configuration of a search system.

As shown in FIG. 1, the search system 100 has a server device 110 and user terminals 170_1 to 170_n used by users 180_1 to 180_n, respectively. The user terminals 170_1 to 170_n are connected to the server device 110 via the network 190 so as to be able to communicate therewith.

A flea market service program is installed in the server device 110 , and the server device 110 functions as a flea market service section 120 by executing the program.

The flea market service unit 120 provides a flea market service for users 180_1 to 180_n of user terminals 170_1 to 170_n connected via the network 190 to provide a place for trading various products.

Here, it is assumed that the flea market service program includes at least a registration program, a product display program, and a search program. By executing these programs, server device 110 functions as flea market service section 120 having product information registration section 130 , product display section 135 , and product information search section 140 .

When the users 180_1 to 180_n send product information from the user terminals 170_1 to 170_n, the product information registration unit 130 receives the product information and stores it in the registration information storage unit 150 . In addition, in this embodiment, a plurality of items of product information stored in the registered information storage unit 150 are collectively referred to as "registered information".

The product information registration unit 130 also extracts a feature vector (a set of a feature amount item and a feature amount value extracted from the image) from the product image included in each item of stored product information, and extracts the feature vector information ( Merchandise) Store in the storage unit 151 .

Furthermore, the product information registration unit 130 registers a specific product image (for example, a product image including a person wearing the product (“wearing image”) other than the product) among the product images included in each item of stored product information. A feature vector is extracted from and stored in the feature vector information (person) storage unit 152 .

The product display unit 135 receives product list requests from the user terminals 170_1 to 170_n, and reads corresponding product information from the registration information 300 in the registration information storage unit 150. FIG. In addition, the product display unit 135 transmits the read product information as a product list to the user terminal (one of the user terminals 170_1 to 170_n) that sent the product list request.

When the product information search unit 140 receives a search request including a query image from the user terminals 170_1 to 170_n, the product information search unit 140 uses the feature vector stored in the feature vector information (person) storage unit 152 to search for the person included in the query image. A process for suppressing the influence of the feature amount of is performed. After that, the product information search unit 140 searches for product images including products similar to the search target included in the query image.

Specifically, the product information search unit 140 obtains a feature vector similar to the feature vector of the post-processing query image from among the feature vectors stored in the feature vector information (product) storage unit 151, thereby obtaining a query image. Search for product images that contain products similar to the search target in the image.

The product information search unit 140 transmits the search result including the searched product image to the user terminal (one of the user terminals 170_1 to 170_n) that sent the search request.

<Hardware Configuration of Server Device>
Next, the hardware configuration of the server device 110 included in the search system 100 will be described. FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device;

As shown in FIG. 2 , the server device 110 has a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 and a RAM (Random Access Memory) 203 . The server device 110 also has a GPU (Graphics Processing Unit) 204 . Note that processors (processing circuits, processing circuits) such as the CPU 201 and GPU 204 and memories such as the ROM 202 and RAM 203 form a so-called computer.

Further, the server device 110 has an auxiliary storage device 205 , a display device 206 , an operation device 207 , an I/F (Interface) device 208 and a drive device 209 . Each piece of hardware of the server device 110 is interconnected via a bus 210 . Further, each hardware shown in FIG. 2 is merely an example, and it goes without saying that the server apparatus 110 may be configured by partially changing (for example, removing the GPU 204).

The CPU 201 is an arithmetic device that executes various programs (for example, a flea market service program, etc.) installed in the auxiliary storage device 205 .

A ROM 202 is a non-volatile memory and functions as a main memory. The ROM 202 stores various programs, data, etc. necessary for the CPU 201 to execute various programs installed in the auxiliary storage device 205 . Specifically, the ROM 202 stores boot programs such as BIOS (Basic Input/Output System) and EFI (Extensible Firmware Interface).

A RAM 203 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), and functions as a main storage device. The RAM 203 provides a work area that is expanded when various programs installed in the auxiliary storage device 205 are executed by the CPU 201 .

The GPU 204 is an arithmetic device for image processing, and when the CPU 201 executes the flea market service program, the GPU 204 performs high-speed arithmetic through parallel processing on various image data. Note that the GPU 204 is equipped with an internal memory (GPU memory), and temporarily holds information necessary for parallel processing of various image data.

The auxiliary storage device 205 stores various programs and various data used when the various programs are executed by the CPU 201 . For example, the registration information storage unit 150 , the feature vector information (product) storage unit 151 , and the feature vector information (person) storage unit 152 are implemented in the auxiliary storage device 205 .

The display device 206 is a display device that displays the internal state of the server device 110 . The operation device 207 is an input device used when an administrator of the server device 110 inputs various instructions to the server device 110 . The I/F device 208 is a connection device for connecting to the network 190 and communicating with the user terminals 170_1 to 170_n.

A drive device 209 is a device for setting a recording medium 220 . The recording medium 220 here includes media for optically, electrically or magnetically recording information, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 220 may also include a semiconductor memory or the like that electrically records information, such as a ROM or flash memory.

Various programs to be installed in the auxiliary storage device 205 are installed by, for example, setting the distributed recording medium 220 in the drive device 209 and reading the various programs recorded in the recording medium 220 by the drive device 209. be done. Alternatively, various programs installed in the auxiliary storage device 205 may be installed by being downloaded via the network 190 .

<Specific example of registration information>
Next, a specific example of registration information in the registration information storage unit 150 will be described. FIG. 3 is a diagram showing a specific example of registration information in a registration information storage unit.

As shown in FIG. 3, the registration information 300 is composed of a plurality of item information. Specifically, the registration information 300 includes "product ID", "product attribute information", and "product image" as information items. The registration information 300 is configured by being stored in the registration information storage unit 150 .

For example, the "product ID" stores an identifier for identifying the product among the product information. "Product attribute information" stores information indicating product attributes (for example, product category, product brand, product title, etc.) among product information. "Product image" stores the product image of the product information.

The product images stored in the "product image" include not only the image of the product itself, but also the product image (wearing image) including the person wearing the product.

<Specific example of feature vector information (product)>
Next, a specific example of the feature vector information (product) stored in the feature vector information (product) storage unit 151 will be described. The feature vector information (product) storage unit 151 stores the feature vector extracted from the product image as feature vector information (product) in association with the product information. FIG. 4 is a diagram showing a specific example of feature vector information (product) stored in a feature vector information (product) storage unit.

As shown in FIG. 4, the feature vector information (product) 400 includes, as information items, "product ID", "product attribute information", and "product image", which are information items related to product information. feature vector". "Feature vector" stores a feature vector extracted from the product image stored in "product image".

As described above, a feature vector is a set of a feature amount item and a feature amount value extracted from an image. In the example of FIG. " is extracted. Also, in the example of FIG. 4, the value of the feature amount is shown as the length of the bar graph.

<Specific example of feature vector information (person)>
Next, a specific example of the feature vector information (person) stored in the feature vector information (person) storage unit 152 will be described. The feature vector information (person) storage unit 152 associates the feature vector extracted from the wearing image with the product information and stores it as feature vector information (person), based on the feature vector extracted from the wearing image. Save the calculated representative vector.

FIG. 5 is a diagram showing a specific example of the feature vector information (person) and the representative vector stored in the feature vector information (person) storage unit.

As shown in FIG. 5, the feature vector information (person) 500 includes "product ID", "product attribute information", "wearing image", and "feature vector" as information items. In the case of the feature vector information (person) 500, a wearing image selected from product images included in each product information constituting the registration information 300 is stored in the "wearing image", and the corresponding product identifier and product attribute are stored. Information is stored in "product ID" and "product attribute information", respectively. Further, the "feature vector" stores a feature vector extracted from the wearing image stored in the "wearing image". As in FIG. 4, the example of FIG. 5 indicates that "feature 1" to "feature m" have been extracted as feature amount items. Also, in the example of FIG. 5, the value of the feature amount is shown as the length of the bar graph.

The representative vector 510 is calculated based on the feature vector of each wearing image stored in the “feature vector” of the feature vector information (person) 500 . The representative vector 510 is obtained, for example, by calculating the average value of the feature vectors of the wearing images. Alternatively, the representative vector 510 is obtained by calculating the median value of the feature vectors of the wearing images. Alternatively, the representative vector 510 is obtained by integerizing the feature vector of each wearing image and calculating the mode.

<Functional configuration of the product information registration unit>
Next, the functional configuration of the product information registration unit 130 included in the flea market service unit 120 of the server device 110 will be described. FIG. 6 is a diagram illustrating an example of a functional configuration of a product information registration unit; As shown in FIG. 6 , product information registration unit 130 has product information acquisition unit 610 , product information (wearing image) selection unit 620 , image feature extraction unit 630 , person feature calculation unit 640 , and product information readout unit 650 .

When the user terminals 170_1 to 170_n send product information from the user terminals 170_1 to 170_n in order for the users 180_1 to 180_n to exhibit products, the product information acquisition unit 610 stores the product information in the registration information storage unit 150 each time it receives the product information. Thereby, the registration information 300 in the registration information storage unit 150 is updated each time the product information acquisition unit 610 receives the product information.

Product information (wearing image) selection unit 620 selects and reads product information including a wearing image from among the product information constituting registration information 300 in registration information storage unit 150 , and notifies image feature extraction unit 630 of the product information. . The product information (wearing image) selection unit 620 selects the product image including the wearing image, for example, at the timing when the product information including the wearing image is registered as the registration information 300 in the registration information storage unit 150 to a certain extent. shall be read as

In this case, the image feature extraction unit 630 extracts a feature vector from the wearing image included in the product information notified by the product information (wearing image) selection unit 620, associates it with the product information, and outputs it to the person feature calculation unit 640. Notice.

Personal characteristic calculation section 640 calculates representative vector 510 based on the characteristic vector associated with each product information notified from personal characteristic calculation section 640 . Further, the person feature calculation unit 640 stores the product information associated with the feature vector extracted from the wearing image as the feature vector information (person) 500 together with the representative vector 510 in the feature vector information (person) storage unit 152 . do.

The product information reading unit 650 reads the registration information 300 in the registration information storage unit 150 and notifies the image feature extraction unit 630 of it.

In this case, the image feature extractor 630 extracts a feature vector from the product image included in each product information constituting the registration information 300 notified from the product information readout unit 650 . The image feature extraction unit 630 also associates the extracted feature vector with the product information, and stores the feature vector information (product) 400 in the feature vector information (product) storage unit 151 .

<Functional configuration of product display unit and product information search unit>
Next, functional configurations of the product display unit 135 and the product information search unit 140 included in the flea market service unit 120 of the server device 110 will be described. FIG. 7 is a diagram showing an example of functional configurations of a product display unit and a product information search unit. As shown in FIG. 7 , the product display unit 135 has a range identification unit 701 and a list generation unit 702 .

The range specifying unit 701 receives product list requests from the user terminals 170_1 to 170_n, and specifies the range of products to be listed based on the received product list requests. The range of products to be displayed in a list indicates, for example, products having a predetermined category as product attribute information, products for which product information is stored for a predetermined period, and the like.

The list generation unit 702 reads the product information of the products included in the specified range from the registration information 300 in the registration information storage unit 150 . In addition, the list generation unit 702 transmits the read product information as a product list to the user terminal (one of the user terminals 170_1 to 170_n) that sent the product list request.

7, the product information search unit 140 includes a query image acquisition unit 710, a search object detection unit 720, an image feature extraction unit 630, an image feature conversion unit 740, a similar vector calculation unit 750, and a search result output unit. It has a part 760 .

The query image acquisition unit 710 is an example of an acquisition unit, receives search requests from the user terminals 170_1 to 170_n, and acquires query images included in the received search requests.

The search target object detection unit 720 detects a region including a product to be searched from the query image as a search target region. However, the function of the search object detection unit 720 is not essential, and the product information search unit 140 does not have to execute the search object detection unit 720 .

The image feature extraction unit 630 extracts a feature vector from the search target area detected by the search target object detection unit 720 and notifies the image feature conversion unit 740 of the feature vector.

The image feature conversion unit 740 reads out the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector notified from the image feature extraction unit 630. Calculate the feature vector of . Specifically, the image feature conversion unit 740 subtracts the value of the feature amount of the representative vector 510 from the value of the feature amount of the feature vector for each item of the feature amount, thereby calculating the feature vector after the subtraction. As a result, the feature vector after the subtraction becomes a feature vector in which the influence of the person's feature amount is suppressed.

Further, the image feature conversion unit 740 converts the feature vector after subtraction by zeroing the negative values included in the feature vector after subtraction, and calculates the feature vector after conversion. Specifically, the image feature conversion unit 740 converts the value of the feature amount to a zero value for the item of the feature amount that has become a negative value due to the subtraction among the item of each feature amount. As a result, the feature vector after conversion becomes a feature vector that does not contain negative values. Although the subtraction is performed to suppress the influence of the person's feature amount, if the effect of the person's feature amount appears as a negative value as a result of the subtraction, it does not mean that the influence of the person's feature amount has been suppressed. , the image feature conversion unit 740 appropriately suppresses the influence of the person's feature amount by converting to a zero value.

The similarity vector calculation unit 750 is an example of a search unit, reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151, and finds similarity to the converted feature vector notified from the image feature conversion unit 740. Find feature vectors that Further, the similarity vector calculation unit 750 notifies the searched feature vector to the search result output unit 760 .

The search result output unit 760 is an example of an output unit, and outputs the product information associated with the feature vector notified from the similarity vector calculation unit 750 as the search result to the user terminal (user terminal 170_1 to user terminal 170_1) that transmitted the search request. 170_n).

<Product information registration process flow>
Next, the flow of product information registration processing in the search system 100 will be described. FIG. 8 is a sequence diagram showing the flow of product information registration processing in the search system.

In step S801, the image feature extraction unit 630 included in the product information registration unit 130 of the server device 110 uses a learning model (feature vector calculation model; details will be described later) for machine learning to extract feature vectors. Get training images. As the learning image, for example, a product image or the like included in a part of product information of a plurality of product information constituting the registration information 300 in the registration information storage unit 150 is used. Note that images to be used for learning images are not limited to product images, and may be any image that can be downloaded on the network 190, for example.

In step S802, the image feature extraction unit 630 machine-learns a feature vector calculation model using the acquired learning image.

In steps S810_1 to S810_3, the user terminals 170_1 to 170_n accept input of product information about products to be exhibited from the users 180_1 to 180_n, and transmit the input to the server device 110. FIG.

In steps S811_1 to S811_3, the product information acquisition unit 610 included in the product information registration unit 130 of the server device 110 sequentially stores the received product information (product images of products exhibited by the users 180_1 to 180_n) as registered information. Stored in unit 150 . Thereby, the registration information 300 is configured in the registration information storage unit 150 .

In step S820, product information (wearing image) selection unit 620, image feature extraction unit 630, and person feature calculation unit 640 included in product information registration unit 130 of server device 110 perform feature vector information (person) and representative vector generation processing. to run. As a result, the feature vector information (person) storage unit 152 stores the feature vector information (person) 500 and the representative vector 510 .

In step S830_1, the product information reading unit 650 and the image feature extraction unit 630 execute feature vector information (product) generation processing. As a result, the feature vector information (product) 400 is stored in the feature vector information (product) storage unit 151 .

In steps S810_4 to S810_6, the user terminals 170_1 to 170_n accept input of product information from the users 180_1 to 180_n and transmit it to the server device 110. FIG.

In steps S811_4 to S811_6, the product information acquisition unit 610 included in the product information registration unit 130 of the server device 110 sequentially stores the received product information in the registered information storage unit 150. FIG. Thereby, the registration information 300 in the registration information storage unit 150 is updated.

In step S830_2, the product information reading unit 650 and the image feature extraction unit 630 execute feature vector information (product) generation processing. As a result, the feature vector information (product) 400 in the feature vector information (product) storage unit 151 is updated. Thereafter, similar processing is repeated in steps S810_7 to S810_9 and steps S811_7 to S811_9.

In this way, the retrieval system 100 first executes the feature vector information (person) and representative vector generation process (step S820) once to generate the representative vector 510. FIG. Thereafter, in the search system 100, each time the registration information 300 is updated multiple times (or each time a predetermined period of time elapses), the feature vector information (product) generation process is executed to update the feature vector information (product) 400. do.

<Product information display/search process flow>
Next, the flow of product information display/search processing in the search system 100 will be described. FIG. 9 is a sequence diagram showing the flow of product information display/search processing in the search system.

In step S900_1, the user terminals 170_1 to 170_n receive product list requests from the users 180_1 to 180_n and transmit them to the server device 110. FIG.

In step S910_1, the product display unit 135 of the server device 110 executes product list display processing, and transmits the product list to the user terminal (any of the user terminals 170_1 to 170_n) that has transmitted the product list request.

In step S900_1, the user terminals 170_1 to 170_n receive and display the product list.

In step S900_1, the user terminals 170_1 to 170_n receive search requests based on the operations performed by the users 180_1 to 180_n in response to displaying the product list, and transmit the requests to the server device 110. FIG.

In step S930_1, the product information search unit 140 of the server device 110 executes product information search processing, and transmits the search result to the user terminal (any of the user terminals 170_1 to 170_n) that sent the search request.

In step S900_1, the user terminals 170_1-170_n receive and display the search results. Thereafter, in the search system 100, every time the user terminals 170_1 to 170_n receive a product list request, the server device 110 executes the product list display process and displays the product list on the user terminals 170_1 to 170_n. In the search system 100, the server device 110 executes product information search processing and displays search results on the user terminals 170_1 to 170_n each time the user terminals 170_1 to 170_n receive a search request.

Note that the product selection process, product list display process, and product information search process shown in FIG. 9 are executed in parallel with the processes after step S810_1 of the product information registration process shown in FIG.

<Details of each step of the product information registration process>
Next, each process included in the product information registration process shown in FIG. 8 (here, feature vector calculation model learning process, feature vector information (person) and representative vector generation process, feature vector information (product) generation process) will be described in detail.

(1) Details of Feature Vector Calculation Model Learning Processing (Step S802) First, details of the feature vector calculation model learning processing (Step S802) will be described. FIG. 10 is a diagram for explaining the learning process of the feature vector calculation model. The image feature extraction unit 630 reads out the learning image and inputs it to the feature vector calculation model 1000 . When a product image included in part of the product information is used as the learning image, the image feature extraction unit 630 reads out the product image from the registered information storage unit 150 as the learning image before the learning process. , is input to the feature vector calculation model 1000 .

Product attribute information such as a corresponding category, brand, pattern, etc., is associated in advance with the learning image. Perform machine learning of the model parameters so that they are output. Note that the feature vector calculation model 1000 is assumed to be configured by a neural network (for example, a convolutional neural network) or the like.

As a result, the image feature extraction unit 630 generates a trained model for the feature vector calculation model 1000 . The image feature extraction unit 630 extracts feature vectors using a learned model of the feature vector calculation model 1000 .

(2) Details of Feature Vector Information (Person) and Representative Vector Generation Processing (Step S820) Next, details of the feature vector information (person) and representative vector generation processing (Step S820) will be described. FIG. 11 is a flowchart showing the flow of feature vector information (person) and representative vector generation processing.

As shown in FIG. 11, in step S1101, the product information (wearing image) selection unit 620 selects product information including a wearing image from the product information constituting the registered information 300 in the registered information storage unit 150. Read out and acquire the wearing image.

In step S1102, the image feature extraction unit 630 extracts feature vectors using the learned model of the feature vector calculation model 1000. FIG. FIG. 12 is a first diagram for explaining processing contents of feature vector information (person) and representative vector generation processing.

As shown in FIG. 12 , the image feature extraction unit 630 sequentially extracts the worn image from the learned model 1200 (a learned model in which up to the k-th layer of the learned model for the feature vector calculation model 1000 can be executed). to enter. As a result, the image feature extraction unit 630 sequentially extracts feature vectors from the k-th layer. In the k-th layer, which is the intermediate layer of the trained model, features that are easy for humans to recognize are calculated. Therefore, by visualizing the k-th layer and calculating the feature vectors, extract feature vectors that are easy for humans to recognize. Because you can. It should be noted that the feature vector extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value. When calculating between feature vectors, if a negative value is included in the feature value, there will be a case where the negative value is subtracted. In such a case, the feature value will swing to the positive side. Because it becomes In other words, this is to avoid a situation in which the influence of the person's feature amount is not suppressed despite the subtraction.

In step S<b>1103 , the person feature calculation unit 640 calculates the representative vector 510 . For example, the person feature calculator 640 calculates the representative vector 510 by calculating the average value of each feature vector extracted from each wearing image (see FIG. 12). Note that the method of calculating the representative vector 510 is not limited to this. For example, the feature vector of the product image is subtracted from the feature vector of each wearing image, the negative values are made non-negative, and then the average value is calculated. , the representative vector 510 may be calculated. According to this calculation method, there is an advantage that the product image is not subtracted too much compared to the case where the representative vector is calculated directly from the feature vector of the wearing image.

In step S<b>1104 , the person feature calculation unit 640 generates feature vector information (person) 500 by associating the read product information with the extracted feature vector, and stores the generated feature vector information (person) 500 in the feature vector information (person) storage unit 152 . do. In addition, the person feature calculation unit 640 stores the calculated representative vector in the feature vector information (person) storage unit 152 .

(3) Details of Feature Vector Information (Product) Generation Processing (Step S830_1, etc.) Next, details of the feature vector information (product) generation processing (Step S830_1, etc.) will be described. FIG. 13 is a flowchart showing the flow of feature vector information (product) generation processing.

As shown in FIG. 13, in step S1301, the product information reading unit 650 reads the registered information 300 in the registered information storage unit 150, and acquires product images included in each item of product information.

In step S1302, the image feature extraction unit 630 uses the trained model 1200 to extract feature vectors from the product image. FIG. 14 is a diagram for explaining the processing contents of the feature vector information (product) generation processing.

As shown in FIG. 14, the image feature extraction unit 630 sequentially extracts product images from a trained model 1200 (a trained model in which up to the k-th layer of the trained model for the feature vector calculation model 1000 is made executable). to enter. As a result, the image feature extraction unit 630 sequentially extracts feature vectors 1410_1 to 1410_p from the k-th layer. Note that, as described above, the feature vectors 1410_1 to 1410_p extracted from the k-th layer are configured so that the value of each feature amount is a non-negative value.

In step S<b>1303 , the image feature extraction unit 630 generates feature vector information (product) 400 by associating the read product information with the extracted feature vector, and stores the generated feature vector information (product) 400 in the feature vector information (product) storage unit 151 . do.

<Details of each process of product information display/search processing>
Next, details of each process included in the product information display/search process shown in FIG. 9 (here, product selection process, product list display process, and product information search process) will be described.

(1) Details of product list display processing (step S910_1, etc.) and product information search processing (step S930_1, etc.) First, details of product list display processing (step S910_1, etc.) and product information search processing (step S930_1, etc.) will be described. .

(1-1) Flow of product list display processing (step S910_1, etc.) and product information search processing (step S930_1, etc.) FIG. 15 is a flow chart showing the flow of product list display and product information search processing. As shown in FIG. 15, in step S1501, the range identifying unit 701 receives product list requests from the user terminals 170_1 to 170_n, and identifies the range of products to be listed based on the received product list requests.

In step S<b>1502 , list generation unit 702 reads product information of products included in the specified range from registration information 300 in registration information storage unit 150 . In addition, the list generation unit 702 transmits the read product information as a product list to the user terminal that has transmitted the product list request.

In step S1503, the query image acquisition unit 710 receives search requests from the user terminals 170_1 to 170_n and acquires query images included in the received search requests.

In step S<b>1504 , the search target object detection unit 720 detects a region including the product to be searched as a search target region from the acquired query image.

In step S1505, the image feature extraction unit 630 extracts feature vectors from the search target area of the acquired query image. FIG. 16 is a diagram for explaining the details of the product information search process. As shown in FIG. 16, the image feature extraction unit 630 extracts the search target region detected from the query image by the trained model 1200 (the learned model for the feature vector calculation model 1000, which is made executable up to the k-th layer). trained model). As a result, the image feature extraction unit 630 extracts the feature vector 1600 of the search target area from the k-th layer. Note that, as described above, the feature vector 1600 extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value.

In step S1506, the image feature conversion unit 740 reads the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector 1600 of the search target region extracted in step S1505. do. Thereby, the image feature conversion unit 740 calculates a feature vector after subtraction.

In step S1507, the image feature transforming unit 740 transforms negative feature amounts into zero values (non-negative) for the feature vectors after the subtraction. Thereby, the image feature conversion unit 740 calculates a feature vector after conversion.

In step S1508, the similarity vector calculation unit 750 reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151, and searches for a feature vector similar to the converted feature vector.

In step S<b>1509 , the search result output unit 760 identifies product information associated with the feature vector searched by the similarity vector calculation unit 750 in the feature vector information (product) 400 . In addition, the search result output unit 760 transmits the specified product information as a search result to the user terminal (any of the user terminals 170_1 to 170_n) that sent the search request.

(1-2) Specific Example of Product Information Search Processing (Step S930_1, etc.) Next, a specific example of the product information search processing (step S930_1, etc.) will be described. FIG. 17 is a diagram showing a specific example of product information search processing. In each feature vector shown in FIG. 17, the horizontal axis indicates the item of the feature amount, and the vertical axis indicates the value of the feature amount.

The image feature conversion unit 740 subtracts the representative vector 510 from the feature vector 1600 of the search target area to calculate a feature vector 1701 after subtraction. As a result, in the feature vector 1701 after subtraction, the feature amount of the search target is weighted more than the feature amount of the non-search target (the influence of the feature amount of the person included in the query image can be suppressed). .

Here, in the image feature conversion unit 740, since the value of the feature amount is subtracted for each feature amount item, the value of the feature amount after subtraction may be a negative value depending on the item of the feature amount ( See dotted line area 1702).

The image feature conversion unit 740 converts the value of the feature amount to zero for the item whose feature amount value is a negative value in the feature vector 1701 after the subtraction (see the dotted line area 1704). Accordingly, the image feature conversion unit 740 calculates a feature vector 1703 after conversion.

The similarity vector calculation unit 750 selects a product having a high degree of similarity with the post-conversion feature vector 1703 from among the product image feature vectors 1410_1 to 1410_p included in each product information read from the feature vector information (product) storage unit 151. Search image feature vectors.

Note that the similarity vector calculation unit 750 may search for feature vectors of a plurality of product images whose similarities are equal to or greater than a predetermined threshold, or may search for the feature vector of the product image with the highest similarity. Due to space limitations, the example in FIG. 17 shows a state in which the feature vector 1410_i of the product image with the highest degree of similarity is retrieved.

The search result output unit 760 reads the product information 1710 associated with the product image feature vector 1410 — i from the feature vector information (product) storage unit 151 . In addition, the search result output unit 760 transmits the read product information 1710 as a search result to the user terminal (one of the user terminals 170_1 to 170_n) that sent the search request.

(1-3) Specific examples of search results of product information search processing (step S930_1, etc.) This will be explained in comparison with images. FIG. 18 is a diagram showing a specific example of search results.

In FIG. 18A, reference numeral 1810 denotes an example of a query image, and reference numeral 1811 denotes a search result (here, 15 individual product images). On the other hand, reference numeral 1812 indicates, as a comparative example, search results (similarly, 15 product images) when general product information search processing is executed for the query image.

As indicated by reference numeral 1811, according to the product information search processing by the product information search unit 140, even when clothes and a person wearing the clothes are included in the query image, clothes similar to the clothes are found. It is possible to search for product images that include a person and that do not include a person.

On the other hand, as indicated by reference numeral 1812, according to general product information retrieval processing, if the query image includes clothes and a person wearing the clothes, product images including the person are retrieved. It will be.

That is, in general product information search processing, product images similar to non-search targets included in the query image are searched.

On the other hand, according to the product information search processing by the product information search unit 140, as described above, the non-search target included in the query image includes products similar to the search target included in the query image. You can search for product images that do not contain.

The same is true for other query images. In FIG. 18B, reference numerals 1820 and 1820' are other examples of query images, and a pattern is reflected in the background of a person wearing clothes.

As indicated by reference numeral 1821, according to the product information search unit 140, as with reference numeral 1811, even if the query image includes clothing to be searched and a person wearing the clothing, It is possible to search for product images that include similar clothing and do not include people. In addition, in the case of reference numeral 1821, even if a pattern is included in the background of the person wearing the clothes, no product image similar to the background pattern is retrieved. This is because the search target object detection unit 720 detects an area including the product to be searched as a search target area. Note that the rectangular area displayed within reference numeral 1820 indicates the search target area detected by the search target object detection unit 720 .

On the other hand, reference numeral 1822 indicates, as a comparative example, a search result when the search target object detection unit 720 does not execute the process of detecting the region including the search target product as the search target region.

As indicated by reference numeral 1822, if the search target area detection process is not executed, product images similar to areas other than the search target area included in the query image are retrieved.

That is, when the process of detecting the search target area is not executed, product images similar to the non-search target included in the query image are searched.

On the other hand, according to the product information search processing by the product information search unit 140, as described above, the non-search target included in the query image includes products similar to the search target included in the query image. You can search for product images that do not contain.

(2) Details of product selection processing (step S900_1, etc.) Next, details of the product selection processing (step S900_1, etc.) will be described.

(2-1) Flow of product selection processing (step S900_1, etc.) and screen example 1
First, the flow of product selection processing (step S900_1, etc.) will be described. FIG. 19 is a first flowchart showing the flow of product selection processing. Here, the flowchart of FIG. 19 will be described with reference to the screen example shown in FIG. Note that FIG. 20 is a first diagram showing an example of the product list screen and the product detail screen.

In step S1901, the user terminal 170_n receives an input of the product range (for example, category) to be displayed as the product list from the user 180_n.

In step S<b>1902 , the user terminal 170 — n generates a product list request including the accepted product range, and transmits the product list request to the server device 110 .

In step S1903, the user terminal 170_n receives the product list from the server device 110 in response to the transmission of the product list request.

In step S1904, the user terminal 170_n displays the received product list on the product list screen. A product list screen 2000 of FIG. 20(a) shows a state in which a product list of dresses is displayed on the user terminal 170_n in response to the user 180_n inputting "dress" as the category.

In step S1905, the user terminal 170_n receives the selection of one desired product from the user 180_n in response to displaying the product list on the product list screen. FIG. 20(a) shows how the product of the product image 2001 is selected by the user 180_n.

In step S1906, the user terminal 170_n transmits a search request to the server device 110 using the product image of one product selected in the product list as a query image. In the example of FIG. 20A, the user terminal 170_n transmits a search request to the server device 110 using the product image 2001 as the query image.

In step S1907, the user terminal 170_n receives the search result from the server apparatus 110 in response to transmitting the search request.

In step S1908, the user terminal 170_n generates and displays a product detail screen. As shown in FIG. 20(b), a product detail screen 2010 displays a product image 2001 of one selected product, detailed information 2011 of one selected product, and a product image 2012 included in the received search results. and are included.

In this way, when one product desired to be purchased is selected on the product list screen 2000, the user terminal 170_n transitions to the product detail screen 2010 and displays the detailed information 2011 of the selected product. . Thereby, the user 180 — n can know the detailed information of one product that the user wishes to purchase on the product detail screen 2010 after the transition.

In addition, by displaying the product image 2012 including products similar to the selected product, the user 180 — n can refer to similar products (related products related to the selected product), and refer to the final product. essentially, you can decide which product to buy. As shown in FIG. 20(b), the product details screen 2010 includes a "purchase procedure" button 2013 for moving to the procedure for purchasing the product when the product to be purchased is decided.

In step S1909, the user terminal 170_n determines whether to return to the product list screen 2000 or not. If it is determined in step S1909 to return to the product list screen 2000 (if Yes in step S1909), the process returns to step S1904.

On the other hand, if it is determined in step S1909 not to return to the product list screen 2000 (No in step S1909), the process proceeds to step S1910.

In step S1910, the user terminal 170_n determines whether to proceed to the purchase procedure screen. If the "purchase procedure" button 2013 has not been pressed, it is determined not to proceed to the purchase procedure screen (No in step S1910), and the process returns to step S1905.

In this case, in step S1905, the user terminal 170_n accepts selection of another product (related product). In the example of FIG. 20(b), the selection of the product of any of the product images 2012 displayed as related products is accepted.

On the other hand, if the "purchase procedure" button 2013 has been pressed, it is determined to proceed to the purchase procedure screen (if Yes in step S1910), and the product selection process ends.

(2-2) Screen example 2 of product selection processing (step S900_1, etc.)
Next, another screen example of the product selection process (step S900_1, etc.) will be described. FIG. 21 is a second diagram showing an example of the product list screen and the product detail screen.

The difference from FIG. 20 is that the user 180_n inputs "new arrivals" for the range of products to be displayed as the product list. When "new arrivals" is entered as the range of products to be displayed as a product list, as shown in FIG. Contains product images. A newly arrived product refers to a product whose product information has been stored within a predetermined period of time going back from the current time.

In this way, user terminals 170_1 to 170_n can arbitrarily input the range of products to be displayed as a product list.

20, when a product is selected on the product list screen, the user terminal 170_n transmits a search request to the server device 110 using the product image of the selected product as a query image. . In the example of FIG. 21A, the user terminal 170_n transmits a search request to the server device 110 using the product image 2101 as the query image.

FIG. 21B shows a transition to the product detail screen 2010 by selecting the product image 2101 on the product list screen 2100 . Note that the configuration of the product details screen 2010 shown in FIG. 21B is the same as the configuration of the product details screen 2010 shown in FIG. 20B, so description thereof is omitted here.

<Summary>
As is clear from the above description, in the server device according to the first embodiment,
・Receive a search request sent from a user terminal and acquire a query image.
・If the query image contains both a search target (e.g., clothes) and a non-search target (e.g., person), among multiple product images, include products similar to the search target, and Search for product images that do not include non-search targets.

Thus, according to the server device according to the first embodiment, it is possible to output search results corresponding to search targets in product searches using query images.

Also, in the user terminal according to the first embodiment,
・When one product is selected from the products displayed in the list and the product details screen is displayed, a search request is sent to the server device using the product image of the selected product as a query image. do.
・When the search result is received from the server device in response to the transmission of the search request, the product image included in the received search result is used as the product image of the related product related to the selected product after the transition. Output to the product detail screen.

Thus, according to the user terminal according to the first embodiment, it is possible to determine the product to be finally purchased while referring to related products related to one selected product.

[Second embodiment]
In the above-described first embodiment, a search request is made at the timing when a product is selected on the product list screen, and the search result is displayed at the timing of transition to the product detail screen. The timing of displaying the results is not limited to this.

For example, a related product search button may be provided on the product detail screen after transition, and a search request may be made at the timing when the related product search button is pressed after transition to the product detail screen.

FIG. 22 is a second flowchart showing the flow of product selection processing. Differences from FIG. 19 are steps S2201, S2202, and S2203.

In step S2201, the user terminal 170_n accepts selection of one desired product from the user 180_n in response to displaying the product list on the product list screen. Thereby, the user terminal 170_n generates and displays a product detail screen including the product information of the selected one product.

In step S2202, the user terminal 170_n accepts pressing of a related product search button for requesting search for products similar to the selected product.

In step S2203, the user terminal 170_n displays the product image included in the received search result as a related product on the product detail screen.

In this way, the user 180_n can refer to the related product at any timing after the transition to the product detail screen by enabling the user 180_n to press the related product search button at any time.

[Other embodiments]
In the first and second embodiments described above, each section (query image acquisition section 710 to search result output section 760) of the product information search section 140 is provided on the server device 110 side. However, some functions of the product information search unit 140 may be provided on the user terminals 170_1 to 170_n. For example, the query image acquisition unit 710, the search object detection unit 720, and the image feature extraction unit 630 may be provided on the user terminals 170_1 to 170_n.

As a result, the server apparatus 110 can receive feature vectors from the user terminals 170_1 to 170_n and execute product information search processing. However, in this case, the user terminals 170_1 to 170_n must acquire the learned model 1200 in advance.

Alternatively, the query image acquisition unit 710, search object detection unit 720, image feature extraction unit 630, and image feature conversion unit 740 may be provided on the user terminals 170_1 to 170_n side. As a result, the server apparatus 110 can receive the converted feature vectors from the user terminals 170_1 to 170_n and execute the product information search process. However, in this case, the user terminals 170_1 to 170_n must acquire the trained model 1200 and the representative vector 510 in advance.

Alternatively, all functions of the product display unit 135 and the product information search unit 140 may be provided on the user terminals 170_1 to 170_n. This allows the user terminals 170_1 to 170_n to execute product list display processing and product information search processing. However, in this case, the user terminals 170_1 to 170_n need to acquire the learned model 1200 and the representative vector 510 in advance, and acquire the feature vector information (product) 400 periodically. This is effective when the number of products is not large or when the frequency of product updates is low.

Further, in the first and second embodiments, the feature vector information (person) and representative vector generation processing is executed only once, and the feature vector information (product) generation processing is executed, for example, every time a predetermined time elapses. was explained as being executed on However, the feature vector information (person) and representative vector generation processing may also be configured to be executed each time a predetermined time elapses, similarly to the feature vector information (merchandise) generation processing. Further, the feature vector information (product) generation process may be configured to be executed each time product information is stored and registration information is updated.

Further, in the first and second embodiments described above, the search target is clothes, and the non-search target is the person who wears the clothes. Then, a representative vector is calculated for a wearing image in which the person wears the clothes. However, the search target is not limited to clothing, and may be accessories, accessories, and the like. In this case, the non-search target is the person wearing the accessory or the person holding the accessory. Then, a representative vector is calculated for a wearing image in which the person wears the accessory (or holds the accessory).

In other words, the wearing image includes all images including a person photographed together with the search target, such as wearing clothes, wearing accessories, holding small items, and the like.

Non-search targets are not limited to people. For example, in an image in which a product is placed on a chair, a chair may be a non-search target. Alternatively, in an image in which a product is hung on a hanger, the hanger may be set as a non-search target. However, if these are not to be searched, it is necessary to calculate a representative vector based on an image containing them.

Further, in the first and second embodiments described above, the product ID, product attribute information, product image, etc. are transmitted as search results. You may choose to send one. Information other than the product ID, product attribute information, and product image may be transmitted as the search result.

Further, in the first embodiment, even if the query image includes clothes and a person wearing the clothes, the product information search unit 140 can find clothes similar to the clothes, In addition, it was shown that it is possible to retrieve product images that do not include people. However, the effect of the product information search processing by the product information search unit 140 is not limited to this. Even when the query image contains only clothes (that is, when the query image is not a wearing image), the product information search unit 140 finds a product image that contains clothes similar to the clothes and does not contain a person. It goes without saying that you can search for

In addition, in the first and second embodiments, the product information search unit 140 calculates the post-conversion feature vector so as to suppress the influence of the person's feature amount on the query image. However, the method of calculating the feature vector after conversion is not limited to this, and for example, the feature vector after conversion may be calculated so that the user's tastes and preferences are clearer in the query image. This makes it possible to implement product recommendation processing for the user.

Further, in the first and second embodiments, the case where the target image searched by the product information search unit 140 is a product image has been described. However, the target images searched by the product information search unit 140 are not limited to product images. For example, it may be a scene image, a painting image, or the like. However, in the case of a scene image, for example, in the query image, the post-conversion feature vector is calculated so as to suppress the influence of the feature amounts of utility poles and automobiles. In the case of a painting image, for example, in the query image, the post-conversion feature vector is calculated so as to suppress the influence of the feature amount of the frame shape.

Further, in the first and second embodiments, a case has been described in which product images that include products similar to the search target and do not include non-search targets are searched and output as product images of related products. However, images to be searched are not limited to product images. For example, any image that includes an article similar to the search target and does not include non-search targets may be searched and output as an image of a related article.

It should be noted that the present invention is not limited to the configurations shown here, such as combinations with other elements, to the configurations listed in the above embodiments. These points can be changed without departing from the gist of the present invention, and can be determined appropriately according to the application form.

100: Search system 110: Server device 120: Flea market service unit 130: Product information registration unit 135: Product display unit 140: Product information search unit 150: Registered information storage unit 151: Feature vector information (product) storage unit 152: Features Vector information (person) storage units 170_1 to 170_n: user terminal 300: registration information 400: feature vector information (product)
500: Feature vector information (person)
510 : Representative vector 610 : Product information acquisition unit 620 : Product information (wearing image) selection unit 630 : Image feature extraction unit 640 : Personal feature calculation unit 650 : Product information reading unit 701 : Range identification unit 702 : List generation unit 710 : Query image acquisition unit 720: Search object detection unit 740: Image feature conversion unit 750: Similar vector calculation unit 760: Search result output unit 1000: Feature vector calculation model 1200: Trained models 1410_1 to 1410_p: Product image feature vectors 1600: Feature vector of search target area 1701: Feature vector after subtraction 1703: Feature vector after conversion 2000: Product list screen 2010: Product detail screen 2100: Product list screen

Claims (8)

an acquisition step of acquiring an image of the selected one item as a query image when one item is selected from the listed items and the next screen is displayed;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a retrieving step of retrieving, from among a plurality of images, an image that includes an article similar to the search target and does not include the non-search target;
A search program for causing a computer to execute an output step of outputting the searched image as an image of a related article related to the selected one article on the next screen after transition. The search step includes:
When the query image includes a search target and a person photographed together with the search target, a feature amount obtained by removing the feature amount of the person from the feature amount of the query image and the feature amount of a plurality of images. 2. The search program according to claim 1 , wherein an image including an article similar to the search target and not including a person is searched from among the plurality of images based on . The search step includes:
3. The search program according to claim 2, wherein a search is performed by weighting the feature amount of a search target included in the feature amount of the query image rather than the feature amount of a person included in the feature amount of the query image. The search step includes:
Subtracting the feature vector containing the feature amount of the query image by the feature vector containing the feature amount of the image containing the article and the person photographed with the article,
Non-negative the negative feature values included in the subtracted feature vector,
3. The search program according to claim 2, wherein a feature vector obtained by non-negating the feature quantity of the negative value is used to calculate a similarity with each feature vector including the feature quantity of each of the plurality of images. causing a computer to perform a calculation step of calculating a representative vector based on each feature vector including feature amounts of each of a plurality of images including an article and a person photographed with the article;
The search step includes:
5. The search program according to claim 4, wherein the representative vector is subtracted from the feature vector containing the feature amount of the query image. an acquisition unit that acquires an image of the selected article as a query image when one article is selected from the listed articles and the next screen is displayed;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a search unit that searches a plurality of images for an image that includes an article similar to the search target and does not include the non-search target;
and an output unit that outputs the retrieved image as an image of a related item related to the selected item on the next screen after transition. an acquisition step of acquiring an image of the selected one item as a query image when one item is selected from the listed items and the next screen is displayed;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a retrieving step of retrieving, from among a plurality of images, an image that includes an article similar to the search target and does not include the non-search target;
an output step of outputting the retrieved image as an image of a related item related to the selected one item on the next screen after transition. A search system in which a terminal and a server device are communicably connected,
an acquisition unit that acquires an image of the selected one item as a query image when one item is selected from the listed items and the next screen is displayed;
When the query image includes a search target and a non-search target , based on the feature quantity obtained by removing the feature quantity of the non-search target part from the feature quantity of the query image and the feature quantity of a plurality of images a search unit that searches a plurality of images for an image that includes an article similar to the search target and does not include the non-search target;
and an output unit that outputs the retrieved image as an image of a related item related to the selected one item on the next screen after transition.

Images