JP2020086914A - Search program, server device, search method, and search system - Google Patents

Abstract

To output a search result according to a search object, in image search using a query image.SOLUTION: A search program causes a computer to execute: an input step of inputting a query image; and a search step of searching for an image which includes an article similar to a search object and does not include a non-search object from among a plurality of images when the query image includes the search object and the non-search object.SELECTED DRAWING: Figure 1

Description

The present invention relates to a search program, a server device, a search method, and a search system.

Conventionally, various search methods for searching for a similar image similar to a query image have been proposed (for example, Patent Documents 1 and 2 below).

JP 2001-052175 A JP 2001-319232 A

However, in the case of the above-described search method, both the search target article and the non-search target are included in the query image, and the feature quantity of the non-search target has a higher degree of similarity than the feature quantity of the search target article. There is a problem that an appropriate image cannot be searched when the influence exerted is large.

The present invention has been made in view of the above problems, and an object of the present invention is to output a search result according to a search target in image search using a query image.

According to one embodiment of the present invention, the search program has the following configuration, for example. That is,
An input step of inputting a query image,
When a search target and a non-search target are included in the query image, a search for searching an image that includes an article similar to the search target and does not include the non-search target from the plurality of images Causes the computer to execute the steps.

According to the present invention, in the image search using the query image, it is possible to output the search result according to the search target.

It is a figure showing an example of a system configuration of a search system. It is a figure which shows an example of the hardware constitutions of a server apparatus. It is a figure which shows the specific example of the registration information in a registration information storage part. It is a figure which shows the specific example of the characteristic vector information (product) stored in the characteristic vector information (product) storage part. It is a figure which shows the specific example of the feature vector information (person) and representative vector preserve|saved at the feature vector information (person) preservation|save part. It is a figure showing an example of functional composition of a goods information registration part. It is a figure showing an example of functional composition of a goods information search part. It is a sequence diagram showing a flow of product information registration processing in the search system. It is a sequence diagram which shows the flow of the goods information search process in a search system. It is a figure for explaining a learning process of a model for feature vector calculation. It is a flow chart which shows a flow of feature vector information (person) and representative vector generation processing. It is a 1st figure for demonstrating the processing content of a characteristic vector information (person) and a representative vector generation process. It is a flow chart which shows the flow of feature vector information (product) generation processing. It is a figure for explaining the processing contents of feature vector information (product) generation processing. It is a flow chart which shows a flow of goods information search processing. It is a figure for explaining the processing content of goods information search processing. It is a figure which shows the specific example of a goods information search process. It is a 1st figure which shows an example of the search result displayed on the search screen of a user terminal. It is a 2nd figure which shows an example of the search result displayed on the search screen of a user terminal. It is a 2nd figure for demonstrating the processing content of a characteristic vector information (person) and a representative vector generation process. It is a figure which shows an example of the chat screen of a user terminal.

The details of each embodiment will be described below. In the description of the specification and the drawings according to each embodiment, constituent elements having substantially the same functional configuration will be denoted by the same reference numerals, and redundant description will be omitted.

[First Embodiment]
<1. System configuration of search system>
First, the system configuration of the search system will be described. FIG. 1 is a diagram illustrating an example of a system configuration of a search system.

As illustrated in FIG. 1, the search system 100 includes a server device 110 and user terminals 170_1 to 170_n used by users 180_1 to 180_n, respectively. The user terminals 170_1 to 170_n are communicatively connected to the server device 110 via the network 190.

The flea market service program is installed in the server device 110, and the server device 110 functions as the flea market service unit 120 by executing the program.

The flea market service unit 120 provides a service of providing a place for trading various products to the users 180_1 to 180_n of the user terminals 170_1 to 170_n connected via the network 190.

Here, it is assumed that the flea market service program includes at least a registration program and a search program. Then, by executing these programs, the server device 110 functions as the flea market service unit 120 including the product information registration unit 130 and the product information search unit 140.

When the users 180_1 to 180_n send the product information from the user terminals 170_1 to 170_n in order to put up the product, the product information registration unit 130 receives the product information and stores it in the registration information storage unit 150. In addition, in the present embodiment, the plurality of product information items stored in the registration information storage unit 150 are collectively referred to as “registration information”.

Further, the product information registration unit 130 extracts a “feature vector” (a set of a feature quantity item and a feature quantity value extracted from the image) from the product image included in each of the stored product information to obtain the feature vector. It is stored in the information (product) storage unit 151.

Further, the product information registration unit 130 is a specific product image (for example, a product image including a person who wears the product in addition to the product (“wearing image”)) among the product images included in the stored product information. A feature vector is extracted from the above and stored in the feature vector information (person) storage unit 152.

When the product information search unit 140 receives a search request including a query image from the user terminals 170_1 to 170_n, the product information search unit 140 uses the feature vector stored in the feature vector information (person) storage unit 152 to identify the person included in the query image. The process of suppressing the influence of the feature amount is performed. Then, the product information search unit 140 searches for a product image including a product similar to the search target included in the query image.

Specifically, the product information search unit 140 obtains a feature vector similar to the feature vector of the processed query image from the feature vectors stored in the feature vector information (product) storage unit 151, thereby performing the query. A product image including a product similar to the search target in the image is searched.

The product information search unit 140 transmits the search result including the searched product image to the user terminal (one of the user terminals 170_1 to 170_n) of the transmission source of the search request.

<Hardware configuration of server device>
Next, a hardware configuration of the server device 110 included in the search system 100 will be described. FIG. 2 is a diagram illustrating an example of the hardware configuration of the server device.

As shown in FIG. 2, the server device 110 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203. The server device 110 also includes a GPU (Graphics Processing Unit) 204. A processor (processing circuit, processing circuit, processing circuitry) such as the CPU 201 and the GPU 204 and a memory such as the ROM 202 and the RAM 203 form a so-called computer.

Further, the server device 110 includes an auxiliary storage device 205, a display device 206, an operation device 207, an I/F (Interface) device 208, and a drive device 209. The hardware of the server device 110 is connected to each other via the bus 210. It is needless to say that the hardware illustrated in FIG. 2 is merely an example, and a part of the hardware may be modified (for example, the GPU 204 is removed) to configure the server device 110.

The CPU 201 is a computing device that executes various programs (for example, a flea market service program) installed in the auxiliary storage device 205.

The ROM 202 is a non-volatile memory and functions as a main storage device. The ROM 202 stores various programs and data necessary for the CPU 201 to execute the various programs installed in the auxiliary storage device 205. Specifically, the ROM 202 stores a boot program such as BIOS (Basic Input/Output System) and EFI (Extensible Firmware Interface).

The RAM 203 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), and functions as a main storage device. The RAM 203 provides a work area in which various programs installed in the auxiliary storage device 205 are expanded when being executed by the CPU 201.

The GPU 204 is a computing device for image processing, and when the flea market service program is executed by the CPU 201, performs high-speed computation by parallel processing on various image data. The GPU 204 is equipped with an internal memory (GPU memory), and temporarily holds information necessary for parallel processing of various image data.

The auxiliary storage device 205 stores various programs, various data used when the various programs are executed by the CPU 201, and the like. For example, the registration information storage unit 150, the feature vector information (product) storage unit 151, and the feature vector information (person) storage unit 152 are realized in the auxiliary storage device 205.

The display device 206 is a display device that displays the internal state of the server device 110. The operation device 207 is an input device used by the administrator of the server device 110 to input various instructions to the server device 110. The I/F device 208 is a connection device for connecting to the network 190 and communicating with the user terminals 170_1 to 170_n.

The drive device 209 is a device for setting the recording medium 220. The recording medium 220 mentioned here includes a medium such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like for recording information optically, electrically, or magnetically. In addition, the recording medium 220 may include a semiconductor memory such as a ROM or a flash memory that electrically records information.

The various programs installed in the auxiliary storage device 205 are installed, for example, by setting the distributed recording medium 220 in the drive device 209 and reading the various programs recorded in the recording medium 220 by the drive device 209. To be done. Alternatively, the various programs installed in the auxiliary storage device 205 may be installed by being downloaded via the network 190.

<Specific example of registration information>
Next, a specific example of registration information in the registration information storage unit 150 will be described. FIG. 3 is a diagram showing a specific example of registration information in the registration information storage unit.

As shown in FIG. 3, the registration information 300 is composed of a plurality of product information. Specifically, the registration information 300 includes “product ID”, “product attribute information”, and “product image” as information items, and each of the plurality of product information is divided for each item of the information. The registration information 300 is configured by being stored in the registration information storage unit 150.

For example, in the “product ID”, an identifier for identifying the product in the product information is stored. In the “product attribute information”, information indicating the attribute of the product (for example, the category of the product, the brand of the product, the title of the product, etc.) in the product information is stored. The "product image" stores the product image of the product information.

It should be noted that the product images stored in the “product image” include not only an image of the product alone but also a product image (wearing image) including a person who wears the product.

<Specific example of feature vector information (product)>
Next, a specific example of the feature vector information (product) stored in the feature vector information (product) storage unit 151 will be described. The feature vector information (product) storage unit 151 stores the feature vector extracted from the product image in association with the product information as the feature vector information (product). FIG. 4 is a diagram showing a specific example of the feature vector information (product) stored in the feature vector information (product) storage unit.

As shown in FIG. 4, in the feature vector information (product) 400, in addition to “product ID”, “product attribute information”, “product image”, which are items of information about product information, “ Feature vector" is included. The "feature vector" stores the feature vector extracted from the product image stored in the "product image".

As described above, the feature vector is a set of the feature amount item and the feature amount value extracted from the image. In the example of FIG. 4, the feature amount items are “feature 1” to “feature m”. "Indicates that it has been extracted. In the example of FIG. 4, the value of the feature amount is shown as the length of the bar graph.

<Specific example of feature vector information (person)>
Next, a specific example of the feature vector information (person) stored in the feature vector information (person) storage unit 152 will be described. The feature vector information (person) storage unit 152 stores the feature vector extracted from the wear image as feature vector information (person) in association with the product information, and based on the feature vector extracted from the wear image. Save the calculated representative vector.

FIG. 5 is a diagram showing a specific example of the feature vector information (person) and the representative vector stored in the feature vector information (person) storage unit.

As shown in FIG. 5, the feature vector information (person) 500 includes “product ID”, “product attribute information”, “wearing image”, and “feature vector” as information items. In the case of the feature vector information (person) 500, the wear image selected from the product images included in each product information forming the registration information 300 is stored in the “wear image”, and the corresponding product identifier and product attribute are stored. The information is stored in "product ID" and "product attribute information", respectively. The "feature vector" stores the feature vector extracted from the wear image stored in the "wear image". As with FIG. 4, the example of FIG. 5 shows that “feature 1” to “feature m” are extracted as the feature amount items. In the example of FIG. 5, the value of the feature amount is shown as the length of the bar graph.

The representative vector 510 is calculated based on the feature vector of each wearing image stored in the “feature vector” of the feature vector information (person) 500. The representative vector 510 is obtained, for example, by calculating the average value of the feature vectors of each wearing image. Alternatively, the representative vector 510 is obtained by calculating the median value of the feature vectors of each wearing image. Alternatively, the representative vector 510 is obtained by converting the feature vector of each wearing image into an integer and calculating the mode value.

<Functional configuration of product information registration section>
Next, a functional configuration of the product information registration unit 130 included in the flea market service unit 120 of the server device 110 will be described. FIG. 6 is a diagram illustrating an example of the functional configuration of the product information registration unit. As shown in FIG. 6, the product information registration unit 130 includes a product information acquisition unit 610, a product information (wearing image) selection unit 620, an image feature extraction unit 630, a person feature calculation unit 640, and a product information read unit 650.

When the users 180_1 to 180_n transmit the product information from the user terminals 170_1 to 170_n to sell the product, the product information acquisition unit 610 stores the product information in the registration information storage unit 150 each time the product information is received. Accordingly, the registration information 300 in the registration information storage unit 150 is updated every time the product information acquisition unit 610 receives the product information.

The product information (wearing image) selection unit 620 selects and reads the product information including the wearable image from among the product information items constituting the registration information 300 in the registration information storage unit 150, and notifies the image feature extracting unit 630 of the selected product information. .. The product information (wearing image) selection unit 620 selects the product image including the wearing image at a timing when the product information including the wearing image is registered as the registration information 300 in the registration information storage unit 150 to a certain extent. And read it.

In this case, the image feature extraction unit 630 extracts a feature vector from the wear image included in the product information notified by the product information (wear image) selection unit 620, and associates the feature vector with the product information to the person feature calculation unit 640. Notice.

The personal characteristic calculation unit 640 calculates the representative vector 510 based on the characteristic vector associated with each item of product information notified from the personal characteristic calculation unit 640. In addition, the personal characteristic calculation unit 640 stores the product information associated with the characteristic vector extracted from the worn image as characteristic vector information (person) 500 in the characteristic vector information (person) storage unit 152 together with the representative vector 510. To do.

The product information reading unit 650 reads the registration information 300 in the registration information storage unit 150 and notifies the image feature extraction unit 630.

In this case, the image feature extraction unit 630 extracts a feature vector from the product image included in each item of product information included in the registration information 300 notified by the product information reading unit 650. Further, the image feature extraction unit 630 associates the extracted feature vector with the product information and saves it as feature vector information (product) 400 in the feature vector information (product) storage unit 151.

<Functional configuration of product information search unit>
Next, the functional configuration of the product information search unit 140 included in the flea market service unit 120 of the server device 110 will be described. FIG. 7 is a diagram illustrating an example of the functional configuration of the product information search unit. As shown in FIG. 7, the product information search unit 140 includes a query image acquisition unit 710, a search target object detection unit 720, an image feature extraction unit 630, an image feature conversion unit 740, a similar vector calculation unit 750, and a search result output unit 760. Have.

The query image acquisition unit 710 is an example of an input unit, receives a search request from the user terminals 170_1 to 170_n, and acquires the query image included in the received search request.

The search target detection unit 720 detects a region including a search target product as a search target region from the query image. However, the function of the search target object detection unit 720 is not essential, and the product information search unit 140 may not execute the search target object detection unit 720.

The image feature extraction unit 630 extracts a feature vector from the search target area detected by the search target detection unit 720 and notifies the image feature conversion unit 740.

The image feature conversion unit 740 reads the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector notified from the image feature extraction unit 630, thereby performing the subtraction. The feature vector of is calculated. Specifically, the image feature conversion unit 740 calculates the feature vector after subtraction by subtracting the value of the feature amount of the representative vector 510 from the value of the feature amount of the feature vector for each item of the feature amount. As a result, the feature vector after the subtraction becomes a feature vector in which the influence of the person's feature amount is suppressed.

In addition, the image feature conversion unit 740 converts the feature vector after the subtraction by zeroing the negative value included in the feature vector after the subtraction, and calculates the feature vector after the conversion. Specifically, the image feature conversion unit 740 converts the value of the feature amount into a zero value (non-negative) for the item of the feature amount that becomes a negative value by subtraction among the items of the feature amount. As a result, the converted feature vector becomes a feature vector that does not include a negative value. Although the subtraction was performed to suppress the influence of the person's feature amount, if the result of subtraction shows the influence of the person's feature amount as a negative value, it does not mean that the influence of the person's feature amount is suppressed. The image feature conversion unit 740 appropriately suppresses the influence of the person's feature amount by converting it into a zero value.

The similarity vector calculation unit 750 is an example of a search unit, and reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151 and resembles the converted feature vector notified from the image feature conversion unit 740. Search for the feature vector to be used. Further, the similar vector calculation unit 750 notifies the search result output unit 760 of the searched feature vector.

The search result output unit 760 uses, as a search result, the product information associated with the feature vector notified from the similar vector calculation unit 750 to the user terminal (one of the user terminals 170_1 to 170_n) that is the sender of the search request. Send.

<Flow of product information registration processing>
Next, a flow of product information registration processing in the search system 100 will be described. FIG. 8 is a sequence diagram showing a flow of product information registration processing in the search system.

In step S801, the image feature extraction unit 630 included in the product information registration unit 130 of the server apparatus 110 uses a learning model for extracting a feature vector (feature vector calculation model; details will be described later) when performing machine learning. Acquire a learning image. As the learning image, for example, a product image included in a part of the product information of the plurality of product information configuring the registration information 300 in the registration information storage unit 150 is used. Note that the learning image is not limited to the product image, and may be any image that can be downloaded on the network 190, for example.

In step S802, the image feature extraction unit 630 machine-learns the feature vector calculation model using the acquired learning image.

In steps S810_1 to S810_3, the user terminals 170_1 to 170_n accept input of product information regarding products to be sold from the users 180_1 to 180_n, and transmit the product information to the server apparatus 110.

In steps S811_1 to S811_3, the product information acquisition unit 610 included in the product information registration unit 130 of the server apparatus 110 sequentially stores the received product information (the product information about the products exhibited by the users 180_1 to 180_n) in registration information. It is stored in the unit 150. As a result, the registration information 300 is configured in the registration information storage unit 150.

In step S820, the product information (wearing image) selection unit 620, the image feature extraction unit 630, and the person feature calculation unit 640 included in the product information registration unit 130 of the server device 110 perform the feature vector information (person) and representative vector generation processing. To execute. As a result, the feature vector information (person) storage unit 152 stores the feature vector information (person) 500 and the representative vector 510.

In step S830_1, the product information reading unit 650 and the image feature extraction unit 630 execute a feature vector information (product) generation process. As a result, the feature vector information (product) 400 is stored in the feature vector information (product) storage unit 151.

In steps S810_4 to S810_6, the user terminals 170_1 to 170_n accept input of product information from the users 180_1 to 180_n and transmit the product information to the server device 110.

In steps S811_4 to S811_6, the product information acquisition unit 610 included in the product information registration unit 130 of the server device 110 sequentially stores the received product information in the registration information storage unit 150. As a result, the registration information 300 in the registration information storage unit 150 is updated.

In step S830_2, the product information reading unit 650 and the image feature extraction unit 630 execute a feature vector information (product) generation process. As a result, the feature vector information (product) 400 in the feature vector information (product) storage unit 151 is updated. After that, the same processing is repeated in steps S810_7 to S810_9 and steps S811_7 to S811_9.

Thus, in the search system 100, first, the characteristic vector information (person) and the representative vector generation process (step S820) are executed once to generate the representative vector 510. Thereafter, in the search system 100, the feature vector information (product) generation processing is executed every time the registration information 300 is updated a plurality of times (or each time a predetermined time elapses), and the feature vector information (product) 400 is updated. To do.

<Flow of product information search processing>
Next, a flow of product information search processing in the search system 100 will be described. FIG. 9 is a sequence diagram showing a flow of product information search processing in the search system.

In step S900_1, the user terminals 170_1 to 170_n accept the search request from the users 180_1 to 180_n and transmit the search request to the server apparatus 110.

In step S910_1, the product information search unit 140 of the server device 110 executes the product information search process and sends the search result to the user terminal (one of the user terminals 170_1 to 170_n) of the sender of the search request.

In step S901_1, the user terminals 170_1 to 170_n receive and display the search result. After that, in the search system 100, each time the user terminals 170_1 to 170_n accept a search request, the server device 110 executes the product information search process and displays the search results on the user terminals 170_1 to 170_n.

The product information search process shown in FIG. 9 is assumed to be executed in parallel with the process of step S810_1 and subsequent steps of the product information registration process shown in FIG.

<Details of each process of product information registration processing>
Next, each step included in the product information registration process shown in FIG. 8 (here, learning process of a feature vector calculation model, feature vector information (person) and representative vector generation process, feature vector information (product) generation process) Will be described in detail.

(1) Details of Feature Vector Calculation Model Learning Process (Step S802) First, details of the feature vector calculation model learning process (step S802) will be described. FIG. 10 is a diagram for explaining the learning process of the feature vector calculation model. The image feature extraction unit 630 reads out the learning image and inputs it to the feature vector calculation model 1000. When using a product image included in a part of the product images as the learning image, the image feature extraction unit 630 reads the product information as a learning image from the registration information storage unit 150 before the learning process. , To the feature vector calculation model 1000.

Product attributes information such as corresponding categories, brands, and patterns are associated in advance with the learning image. When the learning image is input, the feature vector calculation model 1000 displays the corresponding product attribute information. Machine learning of model parameters is performed so that they are output. The feature vector calculation model 1000 is assumed to be configured by a neural network (for example, a convolutional neural network) or the like.

As a result, the image feature extraction unit 630 generates a learned model for the feature vector calculation model 1000. The image feature extraction unit 630 extracts the feature vector using the learned model of the feature vector calculation model 1000.

(2) Details of Feature Vector Information (Person) and Representative Vector Generation Process (Step S820) Next, details of the feature vector information (person) and representative vector generation process (step S820) will be described. FIG. 11 is a flowchart showing the flow of the characteristic vector information (person) and representative vector generation processing.

As shown in FIG. 11, in step S1101, the product information (wearing image) selecting unit 620 selects the product information including the wearing image from the product information forming the registration information 300 in the registration information storing unit 150. Read out and acquire the worn image.

In step S1102, the image feature extraction unit 630 extracts a feature vector using the learned model for the feature vector calculation model 1000. FIG. 12 is a first diagram for explaining the processing contents of the characteristic vector information (person) and the representative vector generation processing.

As shown in FIG. 12, the image feature extraction unit 630 sequentially trains the worn image on a learned model 1200 (a learned model capable of executing up to the k-th layer of the learned model for the feature vector calculation model 1000). To enter. As a result, the image feature extraction unit 630 sequentially extracts feature vectors from the kth layer. In the k-th layer, which is the middle layer of the learned model, the feature amount that is easily recognized by humans is calculated. Therefore, by visualizing the k-th layer and calculating the feature vector, the feature vector that is easily recognized by humans is extracted. Because you can do it. Note that the feature vector extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value. When performing a calculation between feature vectors, if a negative value is included as the value of the feature amount, a case of subtracting with the negative value will occur, and in such a case, the value of the feature amount will shift to the positive value side. Because. In other words, this is to avoid a situation in which the influence of the characteristic amount of the person is not suppressed despite the subtraction.

In step S1103, the personal characteristic calculation unit 640 calculates the representative vector 510. The person feature calculation unit 640 calculates the representative vector 510 by calculating the average value of each feature vector extracted from each wearing image (see FIG. 12 ). Note that the method of calculating the representative vector 510 is not limited to this. For example, the feature vector of the product image is subtracted from the feature vector of each wearing image, the negative value is made nonnegative, and then the average value is calculated. Then, the representative vector 510 may be calculated. According to such a calculation method, there is an advantage that the product image is not excessively subtracted as compared with the case where the representative vector is directly calculated from the feature vector of the worn image.

In step S1104, the personal characteristic calculation unit 640 generates the characteristic vector information (person) 500 by associating the extracted product vector with the read product information, and stores the characteristic vector information (person) 500 in the characteristic vector information (person) storage unit 152. To do. Further, the person feature calculation unit 640 stores the calculated representative vector in the feature vector information (person) storage unit 152.

(3) Details of Feature Vector Information (Product) Generation Process (Step S830_1 etc.) Next, details of the feature vector information (product) generation process (Step S830_1 etc.) will be described. FIG. 13 is a flowchart showing the flow of the feature vector information (product) generation processing.

As shown in FIG. 13, in step S1301, the product information reading unit 650 reads the registration information 300 in the registration information storage unit 150 and acquires the product image included in each product information.

In step S1302, the image feature extraction unit 630 uses the learned model 1200 to extract a feature vector from the product image. FIG. 14 is a diagram for explaining the processing content of the characteristic vector information (product) generation processing.

As illustrated in FIG. 14, the image feature extraction unit 630 sequentially learns the product images from a learned model 1200 (a learned model that can execute up to the k-th layer of the learned model of the feature vector calculation model 1000). To enter. As a result, the image feature extraction unit 630 sequentially extracts the feature vectors 1410_1 to 1410_p from the k-th layer. As described above, the feature vectors 1410_1 to 1410_p extracted from the k-th layer are configured such that the value of each feature amount is a non-negative value.

In step S<b>1303, the image feature extraction unit 630 generates the feature vector information (product) 400 by associating the read product information with the extracted feature vector, and saves it in the feature vector information (product) storage unit 151. To do.

<Details of product information search processing>
Next, details of the product information search process (step S910_1 and the like) will be described. FIG. 15 is a flowchart showing the flow of product information search processing. As shown in FIG. 15, in step S1501, the query image acquisition unit 710 receives a search request from the user terminals 170_1 to 170_n and acquires the query image included in the received search request.

In step S1502, the search target detection unit 720 detects a region including the search target product as a search target region from the acquired query image.

In step S1503, the image feature extraction unit 630 extracts a feature vector from the search target region of the acquired query image. FIG. 16 is a diagram for explaining the processing content of the product information search processing. As illustrated in FIG. 16, the image feature extraction unit 630 enables the search target region detected from the query image up to the learned model 1200 (kth layer of the learned model for the feature vector calculation model 1000). Input to (learned model). As a result, the image feature extraction unit 630 extracts the feature vector 1600 of the search target region from the kth layer. As described above, the feature vector 1600 extracted from the k-th layer is configured such that the value of each feature amount is a non-negative value.

In step S1504, the image feature conversion unit 740 reads the representative vector 510 stored in the feature vector information (person) storage unit 152, and subtracts the representative vector 510 from the feature vector 1600 of the search target area extracted in step S1503. To do. Thereby, the image feature conversion unit 740 calculates the feature vector after the subtraction.

In step S1505, the image feature conversion unit 740 converts (non-negativeizes) the negative feature amount of the subtracted feature vector into a zero value. Accordingly, the image feature conversion unit 740 calculates the converted feature vector.

In step S1506, the similar vector calculation unit 750 reads the feature vector information (product) 400 from the feature vector information (product) storage unit 151 and searches for a feature vector similar to the converted feature vector.

In step S1507, the search result output unit 760 identifies, in the feature vector information (product) 400, the product information associated with the feature vector searched by the similarity vector calculation unit 750. Further, the search result output unit 760 transmits the specified product information as a search result to the user terminal (one of the user terminals 170_1 to 170_n) of the transmission source of the search request.

<Specific example of product information search processing>
Next, a specific example of the product information search process will be described. FIG. 17 is a diagram showing a specific example of the product information search process. In each of the feature vectors shown in FIG. 17, the horizontal axis is the item of the feature amount, and the vertical axis is the value of the feature amount.

The image feature conversion unit 740 calculates the feature vector 1701 after subtraction by subtracting the representative vector 510 from the feature vector 1600 of the search target area. As a result, in the feature vector 1701 after subtraction, the feature amount of the search target is weighted more than the feature amount of the non-search target (the influence of the feature amount of the person included in the query image can be suppressed). ..

Here, since the image feature conversion unit 740 subtracts the value of the feature amount for each item of the feature amount, the value of the feature amount after the subtraction may be a negative value depending on the item of the feature amount (( (See dotted area 1702).

The image feature conversion unit 740 converts the value of the feature amount into zero for the item in which the value of the feature amount becomes a negative value in the feature vector 1701 after subtraction (see the dotted line area 1704). As a result, the image feature conversion unit 740 calculates the converted feature vector 1703.

In the similar vector calculation unit 750, a product having a high similarity to the converted feature vector 1703 among the feature vectors 1410_1 to 1410_p of the product images included in each product information read from the feature vector information (product) storage unit 151. Search the image feature vector.

Note that the similarity vector calculation unit 750 may search for a feature vector of a plurality of product images whose degree of similarity is equal to or higher than a predetermined threshold value, or may search for a feature vector of a product image having the highest degree of similarity. The example of FIG. 17 shows a state in which the feature vector 1410_i of the product image having the highest similarity is searched due to space limitations.

The search result output unit 760 reads the product information 1710 associated with the feature vector 1410_i of the product image from the feature vector information (product) storage unit 151. Further, the search result output unit 760 transmits the read product information 1710 to the user terminal (one of the user terminals 170_1 to 170_n) of the transmission source of the search request as the search result.

<Specific example of search result of product information search process>
Next, a specific example of the search result displayed on the search screen of the user terminals 170_1 to 170_n will be described. FIG. 18 is a first diagram showing an example of search results displayed on the search screen of the user terminal.

As shown in FIG. 18, the search screen 1800 includes a “query image read” button 1811 for reading a query image and a query image display field 1801 for displaying the read query image. Further, on the search screen 1800, a “search request” button 1812 for transmitting a search request including a query image, and a search result display field for displaying the search result transmitted from the server device 110 in response to the transmission of the search request. 1821 and are included.

FIG. 18A shows search results (here, 15 product images) when the product information search processing (see FIG. 15) is executed by the product information search unit 140 in the server device 110. It shows the situation.

As shown in FIG. 18A, according to the product information search processing by the product information search unit 140, even when the query image includes clothes and a person wearing the clothes, the product image is similar to the clothes. It is possible to search for a product image that includes clothing that does not include a person.

On the other hand, FIG. 18B shows, as a comparative example, a display of search results when general product information search processing is executed.

As shown in FIG. 18B, according to the general product information search process, when the query image includes the clothes and the person wearing the clothes, the product image including the person is searched. Will be.

That is, in the case of general product information search processing, product images similar to the non-search target included in the query image are searched.

On the other hand, according to the product information search process by the product information search unit 140, as described above, the non-search target that includes the product similar to the search target included in the query image and that is included in the query image is included. You can search for product images that are not included.

FIG. 19 is a second diagram showing an example of search results displayed on the search screen of the user terminal. Note that the configuration of the search screen 1800 is the same as the configuration of the search screen 1800 shown in FIG. 18, and therefore the description of the configuration of the search screen 1800 is omitted here.

The difference from FIG. 18 is the query image displayed in the query image display field 1801. In the case of the query image displayed in the query image display field 1801 of FIG. 19, a pattern appears on the background of a person wearing clothes.

As shown in FIG. 19A, according to the product information search unit 140, as in FIG. 18A, the query image includes the clothes to be searched and the person wearing the clothes. Even in this case, it is possible to search for a product image that includes clothes similar to the clothes and does not include a person. In addition, in the case of FIG. 19A, even if the background of the person wearing the clothes includes a pattern, a product image similar to the background pattern is not searched. This is because the search target detection unit 720 detects a region including the search target product as a search target region. The rectangular area in the query image displayed in the query image display field 1801 indicates the search target area detected by the search target detection unit 720.

On the other hand, in FIG. 19B, as a comparative example, the search result is displayed when the search target object detection unit 720 does not execute the process of detecting the area including the search target product as the search target area. Is shown.

As shown in FIG. 19B, when the process of detecting the search target area is not executed, a product image similar to the area other than the search target area included in the query image is searched.

That is, when the process of detecting the search target area is not executed, the product image similar to the non-search target included in the query image is searched.

On the other hand, according to the product information search process by the product information search unit 140, as described above, the non-search target that includes the product similar to the search target included in the query image and that is included in the query image is included. You can search for product images that are not included.

<Summary>
As is clear from the above description, in the server device according to the first embodiment,
-Receive the search request transmitted from the user terminal and acquire the query image.
When the query image includes both a search target (for example, clothes) and a non-search target (for example, a person), the plurality of product images include a product similar to the search target, and Search for product images that do not include non-search objects.

As a result, according to the server device of the first embodiment, it is possible to output the search result according to the search target in the product search using the query image.

[Second Embodiment]
In the first embodiment, the representative vector is calculated from the feature vectors extracted by selecting the product information including the worn image from the registration information 300 and sequentially inputting the worn image into the learned model 1200. As explained. However, the method of calculating the representative vector is not limited to this.

For example, when a product image including a worn image is selected, the selected product image may be classified into categories, representative vector candidates may be extracted for each classified category, and then the representative vector may be calculated.

FIG. 20 is a second diagram for explaining the processing content of the feature vector information (person) generation processing. The difference from FIG. 12 is that in the case of FIG. 20, the worn images are classified into categories and are sequentially input to the learned model 1200 for each category.

Thereby, for example, the feature vector of category 1 is extracted for the wearing image classified into category 1, and the feature vector of category 2 is extracted for the wearing image classified into category 2. Further, for the worn image classified into the category x, the feature vector of the category x is extracted.

Here, in the second embodiment, the person feature calculator 640 calculates a representative vector candidate for each category based on the feature vector extracted for each category. Further, the person characteristic calculation unit 640 calculates the representative vector 2000 by acquiring the representative vector candidates of each category and calculating the average value.

However, the method of calculating the representative vector 2000 is not limited to this. For example, even if the representative vector 2000 is calculated by subtracting the feature vector of the product image from the feature vector of each wearing image for each category to make the negative value non-negative and then calculating the average value between the categories. Good.

As is clear from the above description, in the server device according to the second embodiment, when the representative vector is calculated based on the worn image, the representative vector candidate is calculated according to the category and then the representative vector is calculated. .

As a result, according to the server device of the second embodiment, the same effect as that of the first embodiment can be enjoyed, and the representative vector can be further optimized.

[Third Embodiment]
In the above first and second embodiments, the search screen 1800 is displayed on the user terminals 170_1 to 170_n when executing the product information search process. However, the product information search process may be executed via a screen (for example, a chat screen) other than the search screen 1800.

FIG. 21 is a diagram showing an example of the chat screen of the user terminal. Note that, in FIG. 21, the length of the screen in the vertical direction is increased in order to fit the display content displayed by scrolling in the vertical direction into one drawing.

As shown in FIG. 21, by inputting the query image 2101 on the chat screen, the question 2102 ("Do you want to search by image?") is displayed. ) Is entered. As a result, a search request including the query image 2101 is transmitted to the server device 110, and the product information search process is executed in the server device 110, so that the search result 2104 is displayed on the chat screen 2100.

As described above, according to the chat screen 2100, the users 180_1 to 180_n can send a search request via chat instead of pressing the “search request” button.

[Other Embodiments]
In the above-described first and second embodiments, each unit of the product information search unit 140 (query image acquisition unit 710 to search result output unit 760) has been described as provided on the server device 110 side. However, some functions of the product information search unit 140 may be provided on the user terminals 170_1 to 170_n side. For example, the query image acquisition unit 710, the search target object detection unit 720, and the image feature extraction unit 630 may be provided on the user terminal 170_1 to 170_n side.

As a result, the server device 110 can receive the feature vector from the user terminals 170_1 to 170_n and execute the product information search process. However, in this case, the user terminals 170_1 to 170_n need to acquire the learned model 1200 in advance.

Alternatively, the query image acquisition unit 710, the search target object detection unit 720, the image feature extraction unit 630, and the image feature conversion unit 740 may be provided on the user terminal 170_1 to 170_n side. As a result, the server device 110 can receive the converted feature vector from the user terminals 170_1 to 170_n and execute the product information search process. However, in this case, the user terminals 170_1 to 170_n need to acquire the learned model 1200 and the representative vector 510 in advance.

Alternatively, all the functions of the product information search unit 140 may be provided on the side of the user terminals 170_1 to 170_n. Thereby, the user terminal 170_1 to 170_n can execute the product information search process. However, in this case, the user terminals 170_1 to 170_n need to acquire the learned model 1200 and the representative vector 510 in advance, and periodically acquire the characteristic vector information (product) 400. This is effective when there are not many product points or when the product update frequency is low.

In the first and second embodiments, the feature vector information (person) and representative vector generation processing is executed only once, and the feature vector information (commodity) generation processing is performed, for example, every time a predetermined time elapses. Described as what to do. However, the characteristic vector information (person) and representative vector generation processing may be executed every time a predetermined time elapses, similarly to the characteristic vector information (product) generation processing. Further, the feature vector information (product) generation process may be configured to be executed each time product information is stored and registration information is updated.

In the first and second embodiments, the search target is clothes and the non-search target is the person wearing the clothes. Then, the representative vector is calculated for the wearing image in which the person wears the clothes. However, the search target is not limited to clothes, and may be accessories, small items, or the like. In this case, the non-search target is the person wearing the accessory or the person holding the accessory. Then, the representative vector is calculated for the wearing image in which the person wears the accessory (or holds the accessory).

That is, it is assumed that the worn image includes all images including a person photographed together with the search target, such as wearing clothes, wearing accessories, and holding small items.

The non-search target is not limited to a person, and for example, in an image in which a product is placed on a chair, the chair may be the non-search target. Alternatively, in an image in which a product has a hanger, the hanger may be a non-search target. However, when these are not searched, the representative vector needs to be calculated based on the image including them.

In the first and second embodiments, the product ID, the product attribute information, the product image, and the like are transmitted as the search result, but at least one of the product ID, the product attribute information, and the product image is transmitted. One may be transmitted. Further, information other than the product ID, the product attribute information, and the product image may be transmitted as the search result.

Further, in the first embodiment, even when the query image includes the clothes and the person wearing the clothes, the merchandise information search unit 140 includes the clothes similar to the clothes, Moreover, it has been shown that it is possible to retrieve a product image that does not include a person. However, the effect of the product information search processing by the product information search unit 140 is not limited to this. Even when only the clothes are included in the query image (that is, when the query image is not the worn image), the product information search unit 140 includes the clothes similar to the clothes and does not include the person. It goes without saying that you can search for.

In the first and second embodiments, the product information search unit 140 calculates the converted feature vector so as to suppress the influence of the person's feature amount in the query image. However, the method of calculating the converted feature vector is not limited to this, and for example, the converted feature vector may be calculated so that the user's hobbies and tastes become clearer in the query image. As a result, it becomes possible to implement the recommendation processing of products for the user.

Further, in the first and second embodiments, the case where the target image searched by the product information search unit 140 is a product image has been described. However, the target image searched by the product information search unit 140 is not limited to the product image. For example, it may be a scene image or a painting image. However, in the case of a scene image, for example, in the query image, the converted feature vector is calculated so as to suppress the influence of the feature amount of the telephone pole or the automobile. In the case of a painting image, for example, in the query image, the converted feature vector is calculated so as to suppress the influence of the feature amount of the frame shape.

Further, in the above-described first and second embodiments, a case has been described in which a product image that includes products similar to the search target and does not include the non-search target is searched. However, the image to be searched is not limited to the product image, and for example, any image that includes an article similar to the search target and does not include the non-search target may be searched.

It should be noted that the present invention is not limited to the configurations shown here, such as the combination of the configurations described in the above embodiments with other elements. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form.

100: Search system 110: Server device 120: Flea market service unit 130: Product information registration unit 140: Product information search unit 150: Registration information storage unit 151: Feature vector information (product) storage unit 152: Feature vector information (person) Storage units 170_1 to 170_n: User terminal 300: Registration information 400: Feature vector information (product)
500: Feature vector information (person)
510: representative vector 610: product information acquisition unit 620: product information (wear image) selection unit 630: image feature extraction unit 640: person feature calculation unit 650: product information read unit 710: query image acquisition unit 720: search target object detection Unit 740: Image feature conversion unit 750: Similarity vector calculation unit 760: Search result output unit 1000: Feature vector calculation model 1200: Learned models 1410_1 to 1410_p: Product image feature vector 1600: Search target region feature vector 1701: Feature vector after subtraction 1703: Feature vector after conversion 1800: Search screen 2000: Representative vector

Claims (9)

An input step of inputting a query image,
When a search target and a non-search target are included in the query image, a search for searching an image that includes an article similar to the search target and does not include the non-search target from the plurality of images A search program that causes a computer to execute processes. The search step is
An image including an article similar to the search target from the plurality of images when the search target and the person photographed together with the search target are included in the query image, and does not include the person The search program according to claim 1, which searches for. The search step is
The search program according to claim 2, wherein the search target feature quantity included in the query image is weighted more than the feature quantity of the person included in the query image to perform the search. The search step is
A feature vector extracted from the query image is subtracted by a feature vector previously extracted from an image including an article and a person photographed with the article,
Negativeize the negative feature quantity contained in the subtracted feature vector,
The search program according to claim 2, wherein a similarity with the feature vector extracted from each of the plurality of images is calculated using a feature vector obtained by non-negativeizing the feature value having the negative value. Based on the feature vector extracted from each of the plurality of images including the article and the person photographed with the article, a computer is caused to execute a calculation step of calculating a representative vector,
The search step is
The search program according to claim 4, wherein the feature vector extracted from the query image is subtracted by the representative vector. The calculation step is
A plurality of images including the article and a person photographed with the article are classified according to the category of the article, and based on the feature vector extracted from each of the plurality of images classified for each category, for each category. Calculate representative vector candidates,
The search program according to claim 5, wherein the representative vector is calculated based on the representative vector candidates calculated for each category. An input part for inputting a query image,
When a search target and a non-search target are included in the query image, a search for searching an image that includes an article similar to the search target and does not include the non-search target from the plurality of images And a server device having a section. An input step of inputting a query image,
When a search target and a non-search target are included in the query image, a search for searching an image that includes an article similar to the search target and does not include the non-search target from the plurality of images A search method having a process and. A search system in which a terminal and a server device are communicably connected,
An input part for inputting a query image,
When a search target and a non-search target are included in the query image, a search for searching an image that includes an article similar to the search target and does not include the non-search target from the plurality of images Search system with department.